Evaluation in Partnership with Community

[Tom Kelly]

https://aea365.org/blog/cd-tig-week-evaluation-in-partnership-with-...

(This was blog post from AEA365 Blog August 15, 2019)

We are Audrey Jordan (evaluation consultant and executive life coach in Southern California at  [e-mail]) and Tom Kelly (VP Knowledge, Evaluation & Learning at Hawai‘i Community Foundation at [e-mail]) and we used to work together helping create local learning partnerships of evaluators and community residents in cities across the US as part of a 10-year, 10-city community systems change initiative. Local learning partnerships were intended to authentically connect community members with evaluators and data providers in working partnerships to evaluate local community change efforts and analyze data around community conditions. We believed then as we do now that evaluation must be conducted not only in partnership with communities but also that community members need the resources and capacity building to be able to engage genuinely and exercise their autonomy throughout the evaluation process.

As evaluators we are constantly working in and with communities. However, here we were intentionally working in urban communities populated mostly by people of color who not only had not had opportunities to hold decision-making power around research in their communities, many also had very negative past experiences with evaluations and external researchers who had failed in their promises to community or even exploited communities by using data to further exacerbate racist or anti-poor attitudes towards their community.

Lessons Learned:

The most important thing we learned in the success of these partnerships was the ongoing trust, communication, and relationship-building that were required between evaluators of all backgrounds even before the work of collaborative data analysis and sensemaking could begin. Each local learning partnership had to come to a shared agreement about the values and principles that would guide their work. For community members this meant defining their consensus needs and expectations more explicitly, including where their interests and concerns were different from the funder’s and evaluators’. For evaluators, this meant putting more time and resources into real relationship-building, listening, knowledge sharing, and even negotiating around issues such as budget and staffing. Ultimately, the fundamental building block of these partnerships was/is mutual respect, steeped in cultural humility by the evaluators.

Our Denver learning partnership lead the way by co-defining principles that guided the evaluation and learning to take place. Before the evaluation activities were planned and even before the community defined local strategies, they came together and defined with the evaluators their priorities and principles including that all work would contribute to human dignity, antiracism, and the equalization of power. (The full principles appear on p17 of Imagine, Act, Believe).

Cool Tip:

Other groups and communities such as the Harlem Children Zone and the San people of South Africa have also defined codes of conduct and engagement for evaluators and researchers working in their communities.

Rad Resources:

• Sustaining Neighborhood Change: The power of resident leadership, s...
• Imagine, Act, Believe: A framework for learning and results in comm...
• Local Learning Partnership Guidebook
• Equitable Evaluation framework, principles and co-learning

 

The American Evaluation Association is celebrating Community Development TIG Week with our colleagues in the Community Development Topical Interest Group. The contributions all this week to aea365 come from our CD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to [e-mail]. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

[Tom Kelly]

3. How did you manage dynamics between evaluator and communities, and power dynamics amongst community members?

This is a great question that even with respect and professional desire to "help" and "not harm" there is still a lack of recognition of the power and privilege of educated professionals not fully respecting or understanding how to FOLLOW community leadership. First, the communities that did this best, frankly, had to struggle through the long task of working together to define shared values and develop trusting relationships. This was usually bumpy. Dealing constructively with conflict. Reminding each other and holding each other accountable to promises, expectations, but also the shared accountability to real results for the children in the nhood - the goal they all shared. We have lots of examples where a stumble/struggle led to better relationships and trust if real learning took place. Frankly also, there were lots of professional evaluators who had the interest to do this work but not the skills or patience or the ability to hold their egos/professional privilege in check (myself included). We also had incredible experiences of residents stepping up and into influential roles as data translators, accountability coaches. 

[Tom Kelly]

2. What was your definition of engagement? 

I need to invite in my colleague Dr. Audrey Jordan (now in CA) [and the late Dr. Mary Achatz (Westat)]  who worked extensively on putting more intention and definition around what "engagement" meant. I will refer to part of her work from this initiative in this document: Sustaining Neighborhood Change: The Power of Resident Leadership, Social Networks, and Community Mobilization  https://www.issuelab.org/resource/sustaining-neighborhood-change-th...

First, there was no clear and shared definition of what "resident engagement" meant at the beginning of the initiative. In one way, that was okay and allowed for some local definition and approach. It also meant that it could be anything without accountability. Jordan and Achatz and local teams helped to push on the definitions more intentionally and consider voice, participation, advocacy, leadership, and accountability to constituency. 

I am also reminded of Vu Le's trickle-down community engagement: https://nonprofitaf.com/2015/01/are-you-or-your-org-guilty-of-trick...

[Tom Kelly]

(Apologies, @Rituu...way too late in response - I hope everyone is healthy and safe)

1. How did you define the community? Who amongst the community members engaged in evaluation?

This ended up being a fascinating part of the work. First, there was some foundation-driven definition of geographic boundaries at the beginning of the initiative with an initial expectation of a neighborhood with high concentration of poor families with children of about 10-12,000. These were negotiated (in part) with local institutional partners. What we learned as time went on, and with more data and participation by residents, that there were differences (small and large) in how residents defined "place" and how the institutions defined the neighborhood -- this also showed up in the differences among census tracts, political subdistricts, natural places separated by natural and human-made barriers, and history. The community survey asked residents to name and draw the boundary of their neighborhood and their definition (and name) differed across race, tenure in the neighborhood, and history. Most dramatically this occurred in Oakland, CA where depending on the household's history, tenure, race - they called and defined the nhood very differently. This was less the case in places with strong, well-defined "neighborhoods" that also had some political identity (and power) and actual political relevance to the city/county government. In neighborhoods with many newcomers and immigrants, this also showed up as "community" and nhood being less place-based -- and organized around where people congregated for religious service (even if they did not live there), ethnic/cultural business centers, etc. I will never assume the concept of place and nhood ever again. 

[Rituu B Nanda]

Here is a video by IDEV AFDB on participatory evaluation

[Rituu B Nanda]

Thanks Tom. I have more questions.

1. How did you define the community? Who amongst the community members engaged in evaluation?
2. What was your definition of engagement? 
3. How did you manage dynamics between evaluator and communities, and power dynamics amongst community members

[Tom Kelly]

Aloha Rituu and thanks for your questions.

1+2a. These were intentional learning and evaluation partnerships with residents of low-income urban neighborhoods. There also were intentional outreach efforts made to specific groups (eg, single mothers, new immigrants) and ethnic/racial groups. 

2b. Although invitations and outreach were broad and ongoing (with resources for barriers to participation like transportation and child care) there was a challenge in engagement especially for young families with children (TIME). Increasingly, the local learning partnerships relied on the relationships and networks of the group to be the communicators to others who could not participate. Financial reimbursements for time were decided by each group with some deciding to offer $ and others insisting that $ was the wrong incentive. 

3. Although not planned at the outset then (20 years ago), I do think the learning partnerships that spent initial time defining some principles of their work and intent did a much better job at balancing voice and participation with additional efforts made for those usually not heard. I also think a lot was learned about how to communicate and advocate using data in ways that did not "put off" or frighten away people who thought that "data" work was not for them or that they could not be in the driver seat around a research agenda. 

[Rituu B Nanda]

Hi Tom, 

Thanks for posting your blog. i facilitate participation in evaluation and would like to learn from your experience. I had three questions. Thanks!

1. Please would you define what you mean by partnership with communities
2. How do you select who to involve in communities in this partnership?
3. How do you ensure that the most marginalized groups in the community get a voice?