Navigating system change evaluation

At Social Finance, we’re interested in achieving big outcomes. But as we’ve learned about how to make change at scale, it’s become more urgent to ask ourselves how we can know whether we are making progress in changing the system.

In this new white paper we offer some insights and reflections on how to approach evaluating system change. This is the latest in our Changing lives, changing systems series, sharing learnings from practical experience of working to create impact at scale.

We cover four broad themes:

• Why is evaluating system change so hard?
• How should evaluation adapt, and what should it not lose?
• What does system change evaluation look like in practice?
• How might we track progress?

We’d like to acknowledge the System Innovation Initiative, supported by the Rockwool Foundation, among many other partners, practitioners, funders, and evaluators in the field of system change, for their contribution to our thinking.

If you are setting out on a system change journey – or funding one – how should you think about your impact and whether you are making progress? What does it look like in practice to balance some of the newer approaches with some of the ​‘core’ roles of evaluation in checking for results and rigorous causal thinking?

The report explores five things evaluative thinking should do to support system change work: 

1. Embed evaluative thinking into your strategic planning

In system change contexts, it makes less sense to separate evaluative thinking from strategic planning. Expect your strategy to need regular iteration as you learn more about the system and your impact in it, and plan to hold evaluative thinking in productive tension with your strategic and creative vision. 

2. Plan for iteration, but don’t lose your ​‘North Star’

While it is important to expect adaptation and iteration as you learn, it is equally important to have a ​‘north star’ to act as a compass for the work.Articulating what you are trying to achieve and how you think you might achieve it, even loosely and with known (and unknown) knowledge gaps is an essential step for ordering the information you will receive about the system. 

3. Check for impact regularly, but put it in (system) context

Demonstrating impact remains important in system work. However, to avoid a narrow focus on intended results, it is important to be sensitive to what is happening in the wider system because of, despite, or adjacent to your effort – both good and bad. We advocate for rigorous reflection about what the results mean, what assumptions they are dependent on, and about the interaction between your specific effort and the wider system.

4. Retain rigorous, causal thinking but be sensitive to emergent outcomes 

Part of making a judgment about the value of a given piece is establishing not just what happened, but why it happened, and what (combination of things) might have caused it. Causal thinking in system change work should be tempered with a sensitivity to emergent outcomes.

5. Treat your strategy as a hypothesis and learn with direction

Although it is tempting to think that good planning and analysis will set up you up with the ​‘right’ strategy, and allow you to transition from ​‘planning’ to ​‘doing’, thinking evaluatively about system change efforts demands learning and iteration as the status quo.