Why system change work poses challenges for evaluation

November 2023 update: since writing this blog we’ve published a new white paper, Navigating system change evaluation, which builds upon the themes discussed below.

At Social Finance, we’re interested in work that makes big, lasting change for people and society. These projects often demand deep change to the way our social and economic systems work, and tend to look and feel different to efforts with more bounded objectives. Yet commissioners and funders often expect to measure their impact in conventional or ​‘off the shelf’ ways.

Compare a six-month local work mentoring programme with a UK wide initiative aiming to reshape the labour market to make it more accessible. With the former, we know the object of evaluation (the mentoring service), the timeframe (six months) and the domain of impact (service beneficiaries in the local area in which the service operates). In systems change approaches like the latter, the effort to change a system can take many shapes — a network of aligned actors, a catalytic pilot venture or a portfolio of connected interventions — it takes place over a longer, or undefined timeframe, and seeks to have impact on a whole system. 

In our experience working to change systems, evaluating such system change efforts can be challenging. This is not necessarily because we don’t have the right approaches, but because people have come to expect evaluations to look and feel a certain way: a clearly defined intervention, quantifiable impact metrics that are observable in the short-term, experimental methods that control variables to isolate causal pathways. These are hard to get in systems change work. 

We reflect here on five reasons why it’s hard to evaluate a system change effort in these more familiar ways.

1. Systems don’t change in the timescale of a two-year funding round, or term in office. 

In fact, it’s more likely to be decades before you see lasting shifts in the way the system operates if you’re trying to affect large scale social change. The more ambitious and wide-reaching your goals are for changing a system, the less likely it is that you’ll be able to see the ultimate fruits of your labour in the short term. 

Funders who are used to receiving evaluations that demonstrate impact within the timeframe of a funding round, or a term in office might have to shift their expectations about how long it will take before system change efforts realise their goals, and recognise trade-offs between seeking nearer-term, visible impact, and pursing deeper, longer-term change.

2. System change outcomes are often emergent

This means it’s hard to predict when and how they will come about. Emergence describes when changes happen at a collective level through the interaction of multiple elements in a way that is not predictable from examining the elements individually. For example, journalists and commentators closely tracking the gradual liberalisation of the USSR in the late 1980s did not predict that these individual movements would add up to the sudden collapse of the Berlin Wall on 9 November 1989. Instead, this seems to have been a result of an interaction between many different factors in the system that pushed the system to a sudden tipping point. You can read more about how something similar played out in our work on the Drive partnership here.

When outcomes are emergent, evaluators will likely have had a hard time predicting how close work is to its most significant milestones (even if they’re just around the corner).

3. Causal relationships are murky in the world of system change

It’s hard to track the links between specific inputs or actions, and specific outcomes when doing system work. Due to the sheer volume of interactions, factors, actors and entities in a given social system, and because many of the outcomes you’re seeking are emergent, it’s hard – even impossible – to identify clear and stable causal pathways between the actions and inputs of a system change effort and the outcomes it’s seeking to achieve. It would be hard, for example, to say that the lobbying actions of a specific LGBTQ organisation solely caused marriage equality in the UK, even if they were instrumental. 

Although we often expect evaluations to be definitive about what precisely an intervention can claim credit for, or what specifically caused a given outcome, in system change work, we will need to accept more uncertainty, settle for best guesses and expect explanations that involve combinations of contributing causal factors.

4. The nature of any intervention is likely to change over time and may be hard to define. 

A good system change effort will likely adapt, change shape, and re-route many times as it learns more about the system it’s intervening in, and as the system itself changes. For example, an effort to radically reduce consumption of animal products may begin as an advocacy organisation, seeking to change people’s attitudes toward animal cruelty, but turn into a food technology accelerator after identifying that developing cheap, tasty and convenient vegan alternatives is a more effective route to change than changing consumer values. 

Alternatively, you might begin as a single organisation, but over time see the effort as made of many partner organisations with complementary goals. For example, the Canadian Partnership for Children’s’ Health and the Environment began as individual community health groups, women’s clinics, and childcare agencies, before coming together as a system change partnership with a common goal and overarching strategy to affect change. 

These examples pose a challenge for evaluation both because they make it harder to know what exactly you’re evaluating – what combination of initiatives, stakeholders, projects, people constitute the effort — and also because they make it harder to hold it constant over the lifetime of the evaluation, meaning its scope might need to radically change half way through.

5. There are multiple perspectives on what constitutes success 

Different actors in a system may perceive success very differently, making it hard to know exactly when you’ve achieved it. In our work on the Changing Futures programme, we found that the definition of success varied by region, by sector and by perspective on the system. For some statutory partners, success was the implementation of jointly commissioned support services that provided better coordination across hospital, mental health, treatment, police and housing services. For some community groups, success was a reduction in the number of people in a given region falling into preventable crisis. And for others again, success was a world where people who felt let down by services and society felt seen and heard. 

Where we might be used to seeing success defined in evaluations in a relatively straightforward way that signals an objective ​‘tools down’ milestone, system change efforts usually demand a richer, potentially more complex definition of success, which may include some tension and contradiction. 

So, are there any advantages?

System change efforts may make evaluators work a little harder for the reasons explored here. But in our experience, tussling with the tricky questions thrown up by evaluating system change efforts often forces us to think more deeply, test our assumptions more rigorously, and engage more empathetically and holistically with the issue at hand. 

In many ways these challenges are welcome provocations that help us engage more deeply with what should surely be the primary questions of any evaluation effort: what is the true value of what we’re doing? And could we be doing it better?

Please let us know if these ideas resonate and what you’re doing to evaluate system change in your own work. You can get in touch at madeline.goldie@socialfinance.org.uk or Jessica.davies@socialfinance.ork.uk