The central question of any pay for success (PFS) project is determining whether the intervention worked. That is, did the project actually help the people it intended to help in the way that it was supposed to help them?
To answer that question, all of the project stakeholders agree on outcomes and an evaluation design before the project launches. The evaluation then determines whether those targets were met. But determining whether the intervention caused the outcomes observed in anevaluation is more complicated.
Because correlation does not equal causation, simply noting that people who received an intervention have better outcomes than people who didn’t does not mean that the intervention is what caused the difference. A good evaluation can tease out what might be motivating the outcomes observed.
Establishing the best evaluation for any intervention requires a focus on five critical problems.The first one is the most straightforward:
- Sample size: If there are too few participants, a large margin of error will lead to inconclusive results or a false negative (the false finding that a program that had a positive effect did not work).
More daunting are problems that cannot be resolved by simply increasing the sample size. If the evaluation does not measure what it intends to measure, because it asks the wrong question or selects the wrong indicator, the evaluation cannot determine any causality. The remaining four problems can be addressed through strong research design:
- Reverse causality: Researchers can often confuse better outcomes with simple differences in who gets into a program compared to those who don’t. For example, most programs require participants to volunteer and their motivation will likely lead to better outcomes whether or not they receive the program. When an exceptionally capable and motivated individual opts to participate in a job training program and laterfinds a job, the success and motivation of that individual likely caused him or her toparticipate in the program, rather than the program causing him or her to be successful and motivated.
- Unobserved effects: Factors which are not immediately recognizable might predict an outcome more than the intervention does. Genetics, past life experience, or other factors not considered when selecting people to receive the intervention might predictsuccessful outcomes regardless of a given intervention. Statistical "models" can only control for characteristics that can be quantified; much of what determines success is difficult to measure, and caution in interpreting results is wise.
- Selection bias: If certain people are more likely to receive the intervention than others, the results could become biased. For example, if a program targets drug-involved offenders for a community-based program rather than prison, the program may accept only those that are perceived as less "dangerous." While some of these factors can beobserved and modeled, such as age, others, like mental health or behavioral issues, maybe unobservable but affect the decision to enroll a client, and thus cause those in the comparison to be substantially different from those in the treatment group.
- External validity: Can a prior finding be generalized to other, or broader, populations? When the population of a previous study is similar to another population, the resultscan be generalized to the new population. But if the two populations are substantially different in demographics, socioeconomic status, or another important factor, the results of the previous evaluation are less likely to translate to the new population.
Acknowledging and understanding these common issues is critical to fully integrating the principles of rigorous evaluation design and building evidence into PFS projects as well as government practice more broadly. While no research can conclusively prove that anintervention caused an outcome, understanding the other factors that could contribute to the outcomes helps researchers to be more confident in the results of any PFS evaluation.
Have a Pay for Success question? Ask our experts here!
As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research. Photo via Shutterstock.
