Merlin Methods for Understanding Program Effects

Written By Isaac Morrison

“Now ordinary people are born forwards in Time, if you understand what I mean, and nearly everything in the world goes forward too. This makes it quite easy for the ordinary people to live, just as it would be easy to join those five dots into a W if you were allowed to look at them forwards, instead of backwards and inside out. But I unfortunately was born at the wrong end of Time, and I have to live backwards from in front, while surrounded by a lot of people living forwards from behind. Some people call it having second sight.”

-Merlin (The Once and Future King)

There was a brief period in my childhood when I went through a “maze” phase; avidly solving page after page of increasingly sophisticated maze books. They provided a welcome opportunity to channel my otherwise unfocused attention into a concrete problem-solving exercise. At some point I discovered that I could solve the mazes much more quickly and easily if I started at the end and worked my way to the beginning.

It makes sense, if you think about it – mazes are generally designed with a “start to finish” perspective in mind.  All of the false passages and divergent paths are arranged to trip-up travel from a single direction so if you’re going backwards from finish line, most of the incorrect choices don’t even seem like choices at all.

During my time in the monitoring and evaluation sector I’ve used a handful of interrelated data collection approaches that leverage a similar sort of 20/20 hindsight to better understand program effects: outcome harvesting, most-significant change (MSC), and reverse tracer studies. There’s plenty of good writing on all three methods (see links at the end of this post) so I’ll just provide a brief summary here:

  • Outcome harvesting – Rather than measuring progress towards pre-determined goals, outcome harvesting first collects information on what has been achieved and then works backwards to determine whether or not a program or intervention contributed to those achievements. The outcomes in question can be intended or unintended, positive or negative, and directly or indirectly related to the intervention. This method seeks to identify or demonstrate a clear, validated connection between the observed outcome and the intervention.

  • Most significant change – The MSC method collects oral accounts of on-the-ground changes reported by beneficiary communities following a specific activity or intervention. These “change stories” do not require an explicit cause and effect relationship to the activity and may be only indirectly or contextually related to the intervention itself. Importantly, however, the stories include explanations of why the interviewees considers those changes to be significant. Once collected, other stakeholder groups are also invited to engage with and interpret the change stories based on their own perspectives of what is or is not significant. The exercise provides insight into complex and ambiguous situations where different stakeholder groups have broadly divergent priorities and definitions of success.

  • Reverse tracer studies – Primarily used in education and employment studies, this is an inversion of the much more commonly used tracer study. Rather than following a cohort group over time to monitor their career path, reverse tracer studies start with a related demographic cross-section and then retrospectively seek to understand what training, education, and experience itineraries brought them to that point.

For me, the appealing thing about all three of these approaches is that instead of starting with the classic evaluation question “Did our activity have the desired effect?” the researcher begins by asking the beneficiary, “Where are you now?” and then works backwards to ask, “How did you get there?” Once those two questions are answered the researcher can pursue a more nuanced and holistic answer to the question, “Did we help, and if so, how?”

This matters because the question “Did our activity have the desired result?” can bias the inquiry towards a focus on the activity rather than the beneficiaries and their context. Although this is not inherently wrong, it creates conditions where researchers can overlook knowledge gaps and overemphasize cause-and-effect attribution or influence. Most importantly it represents a missed opportunity to learn what the intended beneficiaries really need.

 Further Reading

Most Significant Change:

Outcome Harvesting:

Reverse Tracer Studies:

Isaac Morrison
Previous

Choosing the Right Pen for Fieldwork Note-taking: Part One
Next

The Two-pen Method of Fieldwork Note-taking