Lisette Vegas
Urban Institute
Project Associate
Urban Institute
Policy Assitant

The fundamental role of evidence in pay for success

January 16, 2019 - 10:27am

Social science is all about evidence building. When a social program delivers services to improve outcomes for a population, researchers evaluate how well that intervention accomplishes these predetermined goals. Over time, testing hypotheses through evaluation builds bodies of evidence that can improve future service delivery. In pay for success (PFS), a strong evidence base contributes to the development of projects in three major ways:

  1. Evidence guides PFS project scoping: At the outset of a project, PFS stakeholders should have a strong grasp of the target population’s unique needs. Consulting evidence around the prevalence and demographics of the target populations is fundamentally important to the planning process.
  2. Evidence informs PFS program selection: PFS stakeholders should use evidence to guide their selection of interventions that best align with the needs of the target population. For example, Nurse-Family Partnership has been evaluated multiple times over the past few decades, which is one of the reasons the intervention was selected for the South Carolina PFS project.
  3. Evidence influences the terms of the PFS deal: Programs with an existing evidence base bestow greater confidence among stakeholders. Programs that can provide evaluation results, such as the Nurse-Family Partnership, can both help determine and price outcomes for a PFS project.

Assessing the evidence base

As stakeholders select an intervention, they should consider prior evaluations of the intervention and asses on how well they were able to determine causality and external validity. When trying to judge the quality of the evidence, a few key questions are helpful:

  1. Did the evaluation control for self-selection bias?
  2. Did the study develop an appropriate comparison group to measure the outcomes of the intervention?
  3. Are the results of this evaluation generalizable to a broader population that could include more people, different people, or other geographic areas?

If stakeholders, can answer “yes” to these questions, they can be more confident that the intervention can generate desired outcomes. PFS can still be useful, however, even in cases where the evidence base might be thinner.

How PFS can contribute to the evidence base

Repetition of any intervention is crucial, as single results do not prove the program’s effectiveness in one setting. Thus, the fact that evaluation is built into every PFS project ensures that the projects contribute to the evidence base and the generalizability of the program. In some cases, PFS evaluations can both take advantage of existing evidence and build the evidence base.

For example, the evaluation of the Denver Social Impact Bond, which provides affordable housing and wraparound services to support people experiencing chronic homelessness and cycling in and out of jail, is tracking different process measures and outcomes such as housing stability, jail bed days, and time to housing. The evaluation is linking payments to housing stability and jail bed day reduction because the research behind those two outcomes is the most developed. Other outcomes, such as detox use, are of interest, but because the evidence base is less developed, there is more risk in the city linking payments to them. Through programmatic and administrative data, the evaluation is, however, tracking more than just housing stability and jail bed reduction. Due to this, at the end of the evaluation the project will contribute to the evidence base.

Evidence is critical for any PFS project, as it helps practitioners select an intervention and set expected outcomes. Because PFS also requires a rigorous evaluation, the projects themselves can be useful in filling holes where fields need more evidence.

Have a Pay for Success question? Ask our experts here!

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research. Photo via ShutterStock.