Principles for selecting PFS outcome metrics

When designing a new pay for success (PFS) project, one of the most important steps is selecting the outcome metrics by which the project will define success. In other words, what social impact do we want to see as a result of this program, and how will it be measured? This is a strategic decision-making question with clear value beyond just PFS projects. Well-defined outcomes help answer: Why are we implementing this intervention? What, specifically, do we hope to achieve? How will our intervention improve peoples’ lives above and beyond what they’re experiencing now?

PFS outcome metrics are composed of three things: big picture outcomes (e.g. reducing homelessness), outcome targets (e.g. reducing homelessness by 50%), and outcome measures (e.g. reduction in shelter bed days by participants).

As with defining the problem and target population, defining outcome metrics should be the result of a strategic discussion informed by local data. Outcome metrics will vary based on the type of program being used, the target population, and other context-driven specifics. However, there are several broad principles that stakeholders should use to guide them as they select the project’s outcome metrics. These principles are driven by two objectives: (1) producing clear and unambiguous results and (2) ability to reflect actual progress on an issue that is a true priority for the jurisdiction.

Outcome metrics should:

• Be few in number. Having one primary outcome metric is acceptable, as long as it helps provide clarity and focus to the project. Communities can still examine the success of the project in multiple ways by selecting secondary outcomes not tied to payments but still with measurable targets. Stakeholders may wish to also have outcome metrics that can be measured along the way in order to track progress and, as the case may be, warn stakeholders if implementation is off track. These additional outcome metrics help the government track other important impacts while not overly complicating the project.
• Be outcomes, not outputs. Outputs are the direct activities and actions produced by the program, while outcomes are behavioral or other changes experienced by the recipients of the intervention. In other words, an increase in the number of group therapy sessions for a group of formerly incarcerated people is an output, but a decrease in recidivism among those people is an outcome.
• Be measurable. Stakeholders should choose outcome(s) that can be measured through clearly identifiable data sources. For instance, if you want to measure the decrease in recidivism in the above example, be sure the Department of Corrections has and is willing to share data on re-offending.
• Be meaningful. Pick outcome targets that reflect change stakeholders consider meaningful. For example, if chronic homelessness is a widely acknowledged problem, significantly reducing it is a meaningful outcome for the community—but measuring arrests among the chronically homeless, for example, does not tell us if that outcome is being achieved.
• Have an appropriate time horizon. If parties are eager for the project to conclude in five years, the metric should not be one that requires at least seven years to show results. Funders will likely be less interested in projects that take many years to pay off and/or do not offer multiple opportunities for success payments through the use of intermediate outcomes. In existing projects, outcomes are often measured at various intervals during service provision.
• Have a target that is realistic, yet ambitious. If you want to see a reduction in recidivism, what is a reasonable and desirable reduction to aim for? Consult the existing literature and experts, and have a frank discussion among the government, service provider, funder(s), and evaluator to determine what the target should be, acknowledging that each party has incentives that might not be completely aligned (e.g. governments may wish to be very ambitious while funders may wish for a lower target that can more easily be met).  
• Be informed by evidence. If a program has existing evaluations of its efficacy, consider what outcome metrics were used and whether you have the infrastructure to measure the same ones. This data point can help estimate risk and determine outcome pricing. It is also important to consider any differences between your community and the community in which the original evaluation was conducted.
• Be assigned a dollar value. Calculating dollar values to assign the achievement of outcome targets is important both to help governments understand the intervention’s “cashable” benefit (or how much money they are able to save as a result of a successful program) but also because this is tied into the very structure of the PFS deal. Funders are paid at the end of a PFS project on the basis of achievement or non-achievement of these outcomes targets.

Developing strong outcome metrics (outcomes, targets, and measurements) is important not only because it brings clarity and purpose to the PFS project but because it forces governments and other stakeholders to think strategically about what they seek to achieve – a way of thinking that can be applied to non-PFS projects as well. Strong outcome metrics will depend on the context but by following these basic principles, stakeholders should have a better sense of where to start.

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research. Photo via Shutterstock.