Navigating system change evaluation

Published: 8 November 2023

Changing systems takes time and effort. But how do you know if you’re making progress? 

At Social Finance, we’re interested in achieving big outcomes. But as we’ve learned about how to make change at scale, it’s become more urgent to ask ourselves how we can know whether we are making progress in changing the system.

In this new white paper we offer some insights and reflections on how to approach evaluating system change. This is the latest in our Changing lives, changing systems series, sharing learnings from practical experience of working to create impact at scale.

We cover four broad themes:

  • Why is evaluating system change so hard?
  • How should evaluation adapt, and what should it not lose?
  • What does system change evaluation look like in practice?
  • How might we track progress?

We’d like to acknowledge the System Innovation Initiative, supported by the Rockwool Foundation, among many other partners, practitioners, funders, and evaluators in the field of system change, for their contribution to our thinking.

System change evaluation in practice

If you are setting out on a system change journey – or funding one – how should you think about your impact and whether you are making progress? What does it look like in practice to balance some of the newer approaches with some of the core’ roles of evaluation in checking for results and rigorous causal thinking?

The report explores five things evaluative thinking should do to support system change work: 

Two circles overlapping. One has the text 'Strategic and creative thinking' inside, the other, 'Evaluative thinking'.

1. Embed evaluative thinking into your strategic planning

In system change contexts, it makes less sense to separate evaluative thinking from strategic planning. Expect your strategy to need regular iteration as you learn more about the system and your impact in it, and plan to hold evaluative thinking in productive tension with your strategic and creative vision. 

Three sets of arrows in a circular shape, one coloured blue, the other two a faded grey; another arrow points to a red star.

2. Plan for iteration, but don’t lose your North Star’

While it is important to expect adaptation and iteration as you learn, it is equally important to have a north star’ to act as a compass for the work.Articulating what you are trying to achieve and how you think you might achieve it, even loosely and with known (and unknown) knowledge gaps is an essential step for ordering the information you will receive about the system. 

A red circle with the text 'impact of your work' inside. Outside that circle, a white circle, then a grey circle with the word 'system'.

3. Check for impact regularly, but put it in (system) context

Demonstrating impact remains important in system work. However, to avoid a narrow focus on intended results, it is important to be sensitive to what is happening in the wider system because of, despite, or adjacent to your effort – both good and bad. We advocate for rigorous reflection about what the results mean, what assumptions they are dependent on, and about the interaction between your specific effort and the wider system.

A circle, segmented into four parts, labelled: How? What? What else? Why? There are arrows indicating a circular motion around the circle. At the centre of the circle is a star, which is labelled 'hypothesised outcome'. At the top, outside of the circle is another star, labelled Unplanned (relevant) outcome.

4. Retain rigorous, causal thinking but be sensitive to emergent outcomes 

Part of making a judgment about the value of a given piece is establishing not just what happened, but why it happened, and what (combination of things) might have caused it. Causal thinking in system change work should be tempered with a sensitivity to emergent outcomes.

Three angle arrows in a row, each pointing toward a small blue circle.

5. Treat your strategy as a hypothesis and learn with direction

Although it is tempting to think that good planning and analysis will set up you up with the right’ strategy, and allow you to transition from planning’ to doing’, thinking evaluatively about system change efforts demands learning and iteration as the status quo. 

The triangle of system change evaluation

We also consider what changes to look out for to know you’re changing the system. Our triangle of system change evaluation can be thought of as a set of guardrails for evaluative thinking, a structure to order thoughts, and a way of articulating what sort of changes you might need to look for to know whether your system change work is making progress. 

Our triangle of system change evaluation highlights three types of impact: 

A triangle. At the top, an orange star with the words North Star inside it. To the bottom left corner of the triangle, a blue box with the words 'Shifts in the raw materials of the system'. To the bottom right, a pink box with the words 'Building blocks of impact' inside. The words 'mutually reinforcing relationship', with arrows, link the bottom right and bottom left blocks

North Star

  • Population level changes
  • Real outcomes in people’s lives
  • New healthier’ systems

Shifts in the raw materials’ of the system

  • New values and beliefs about what’s possible
  • New culture and behaviours
  • Power shifting to new groups
  • New relationships between parts of the system

Building blocks of impact

  • Supportive policy, legislation, regulations
  • New or changed funding
  • Accountability or incentives shift
  • New public conversation
  • Sector adopts new approaches
  • Widespread solution delivery

Related publications

Stay up to date

Sign up to our mailing list for regular updates.

We'll keep your data secure and won't share it outside of Social Finance, ever. For more details read our privacy policy.