Put understanding first. Then (maybe) measure.

[Silva Ferretti]

There is this knee-jerk reflect about "putting a number on everything". As if only numbers give a seal of credibility and authority. We should start being more vocal in resisting the tyranny of "measures first".

In social development we are grappling with many complex ideas, which can take many forms.

How can we measure them if we do not even understand them?

Too often we are putting measure before understanding, as if measure could tell us all that we need to know.

Not everything that matters can be measured... But, certainly, a lot more could be understood better. Too many time I saw effort in measuring go nowhere and preventing understanding. We need to FIRST capture
dynamics at play, see what changes look like, what is shaping it. Once you improve understanding, you can then measure some aspects of change. You will then able to cherry pick the aspects worth measuring - if need be. But of you start with measuring, then the measure itself will be compromised by the many assumptions made.

I wonder how many millions dollars and how many hours of time have been wasted in seeking to measure things that do not really matter, following the blueprint... resources which should have been invested in understanding.

Some feel that "measures improve accountability". Yes, they can help, but only if there is a clear understanding about what is measured. Otherwise, numbers can easily obfuscate reality (as anyone good at working  with numbers knows!). Accountability is not only about measures. Sharing understanding is a great way to increase accountability and dialogue. Drawing together a map of "how change happens" can generate stronger accountability than a measure. A "rich picture diagram" (as used in soft system analysis, for example) can document what does change look like... What drives it... How do things connect.

The initial rich picture can be just a brainstorming. But as experience accumulates around it, more factors and evidence will be captured, and the pictures can become even richer. Good rich pictures capture understanding. And they reveal the system behind. Understanding complexity requires to appreciate, document systems.

Many participatory tools had grasped this, well before "system theory", "complexity", "adaptive development" started to become fashionable. There is a lot of wisdom in these tools. They can generate accountability, learning.. they can distill evidence. And, if need arises, they can be linked to meaningful measures. 

You can go from understanding to measure. But the opposite, going from a preset measure towards understanding, is not possible. It is like trying to look at a landscape with horse blinders.

Yet these participatory tools, so good at generating understanding, are increasingly eroded.

Should we start insisting for "understanding first" (and only then, and if need be, measure?) rather than accepting the tyranny of "measure first"?

And I do love numbers! But I love numbers which add meaning, which really pin down aspects worth drilling into. Not the numbers and the measures that steal the time and resources that should be devoted to understanding.

[Scott Chaplowe]

Great to see this post, Silva. It resonates well with an article I co-authored, Evaluating Outside the Box, https://socialinnovationsjournal.com/index.php/sij/article/view/704. "Whereas it is often asserted that, 'What gets measured gets done,' another witticism reminds us that, 'Not everything that can be counted counts, and not everything that counts can be counted.' Reality is not a binary concept that can be counted, nor wrapped up into neat, quantitative boxes with key performance indicators (KPIs) that measure whether funded interventions achieve impact. The same attributes that make systems complex make interventions within hard to measure."

[Silva Ferretti]

This Friday (24th March) I will have the privilege to talk about this topic at the EvalForward talks.  It is a nice setup for quick presentations followed by discussion. If you are interested, this is the link https://www.evalforward.org/webinars .

[Margerit Roger]

What a wonderful distinction to make. Thanks for writing this. Some time ago I read a great book called "What Money Can't Buy: The Moral Limits of Markets" by Michael Sandel. He raises questions about how we came to be not just a market economy but a market society. And I think it's the drive for numbers (which is equated to dollars) that has led us down that imbalanced path. I will refer to you when working with clients: let's understand first and then measure what is truly material. 

[Sindhu Nambiath]

Interesting read Silva, thanks for sharing.

Understanding something before learning it,  measuring something to understand and learn, measuring to report ....it all depends upon how and for what we are doing it and how much we want to go. Quiet agree with Hagop and Clara. Now a days mostly a combination of suitable methods are used to understand, know, measure  and learn situations, changes or transformations

[Clara Rodriguez Ribas]

Yet the devil lies in the details- even if existing methods are inadequate, they are the tools we have available to measure, contrast, compare and comprehend the world. If we do not find methods that help us understand realities, then we are stuck in aspirational intentionalities. 

When I rea d your post I thought you were "reacting" in particular to the recent trend in developing composite indexes for everything- and how they seem to continue to fail to fully grasp the realities they are trying to portray. Yet, they are useful tools to advocate for change, compare varying progress towards goals, and move the agenda forward. As inadequate as they may be, they are a step froward from measuring purely numerical values.

[Annemarie Westendorp]

Thanks for sharing Silva. I fully agree with you. Participatory indicator development and m&e processes might be an acceptable way to monitor change and find a balance between quantitative and qualitative approaches?

[Silva Ferretti]

It is not about "quantitative = the narrative" and "qualitative = the numbers"... I feel that this separation is not even meaningful anylonger. Where do you put "a network" for example? The point is that there are structures, systems underpinning our action. Understanding means to "see the system". Over and over I fail to see such systems. And, without that understanding, neither qualitative or quantiative makes any sense.

[Rituu B Nanda]

Brilliant point Laura. Then reporting becomes a checklist, a formality.  So we need to think- who reports to whom!

I have introduced a systematic reflection process in communities and NGOs I work with. To reflect on what they are doing. This triggers them to adapt their strategy.

On Hagop's point- I would add that in my experience individual and collective understanding and sensemaking throughout the process is effective.

[Clara Rodriguez Ribas]

Thanks for this interesting post and thoughts. When reading it I was thinking along the lines of Rituu's response. What are the "numerical measures" you are reacting to? Which numbers are measuring something we are not understanding yet? 

Also, sometimes we collect numbers so that we can have a better understanding of things- not just  to measure change. (which in itself is also a tool for understanding...)

I guess another way to argue your case is the importance of combining quantitative analysis with qualitative ones- numbers. tell us one part of the story, but stories and narratives another part of the picture. This is even more relevant for gender studies- as evidence keeps pointing out we need to go beyond numbers to capture the realities lived by women and girls globally. (For example, recommendations for assessing the extent to which a programme is gender responsive usually indicate the importance of both quantitative and qualitative indicators)

Thanks again for posting. Best

Clara

[Laura Hughston]

Very valid points Silva but there is also an additional dimension to the problem. We don't simply aim to measure or understand change, we seek to measure it in relation to what we do. We don't measure to understand, we measure to report.