George Grob on Qualitative Evaluation for Policy Makers - AEA tips

[Asela Kalugampitiya]

Hi. I am George Grob, an evaluation consultant focusing on policy development and advocacy. During 40 years of Federal service, mostly in the Department of Health and Health and Human Services), I learned that policy makers (Members of Congress and high level executives) are very interested in evaluations. They especially like observations, real life stories, and field reports.

Hot Tips:

#1. Hit the pavement. Some of our more compelling evaluations were based on onsite reviews and discussions with program beneficiaries. For example, in the face of a severe shortage of foster families, state and local agencies began intensive media campaigns to recruit them. That didn’t work. We were asked to find out why not. When we interviewed foster families they told us they joined up because they were impressed with other foster parents they had met. Our report showed that this informal foster parent network could be used as a far more powerful motivating force than the advertisements.

#2. Tell them something they don’t already know. Policy makers and their staff are very well read and constantly in touch with advocates and researchers. It is important to learn where their knowledge black holes are. That’s easy. They will tell you. Work on those. For example, the U.S. Surgeon General was getting worried about wine coolers. She was afraid that kids were being enticed into drinking what seemed like a harmless alcoholic drink that could be a gateway to heavier drinking. She was right. We found that convenience stores near schools were placing wine coolers in fruit juice aisles. Bottle labels obscured the alcoholic content–dark colored backgrounds with only slightly darker and similarly colored ink; small font size. We asked kids to select wine coolers from similarly bottled fruit juices. They couldn’t do it. Our report buttressed the Surgeon General’s campaign to reduce child consumption of wine coolers.

#3. Answer their questions. One of my favorite stories is about the distress of policy makers who noticed that a seemingly disproportionate amount of foster care dollars were being spent on administration instead of foster care payments. They were ready to carve out what they believed was waste and give it to the kids. They asked us to look into it. Our study found that most of what was labeled as “administrative costs” was case work—family studies to determine the best placement of a child and to prepare for court hearings. This was the very heart of the foster care system. The policy community sheathed their knives and looked deeper into the foster care system.

Rad Resource: Grob, G. (2014). Qualitative inquiry for policy makers. In Goodyear, L., Jewiss, J., Usinger, J., & Barela, E. (Eds.), Qualitative inquiry in evaluation: From theory to practice. Jossey-Bass, pp. 55-76.

The American Evaluation Association is celebrating Qualitative Evaluation Week. The contributions all this week to aea365 come from evaluators who do qualitative evaluation. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to [e-mail]. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.