Urban Institute
Distinguished Fellow
Urban Institute
Research Assistant

Tracking additional indicators to understand what works

Series: Using Your Pay for Success Project to Capture Additional Lessons
January 4, 2018 - 1:22pm

Rigorous evaluation of social programs can provide valuable lessons for government. For pay for success projects, which place such evaluation at their core, there are lessons to be learned even if the project does not meet agreed-upon outcomes.

This blog is the first of a three-part series that explores opportunities to learn:

1. what other indicators are important for understanding program quality

2. how the intervention affects the people it served

3. how changes in service delivery procedures during implementation can impact outcomes

This blog describes the first opportunity.

 

While evaluations in a pay for success (PFS) project are primarily used to determine whether the project has achieved the target outcomes on which success payments are based, they can also shed light on other outcomes and various process indicators at low marginal cost.

Repayments are typically contingent on only a small number of outcomes, but others can provide additional insight about what works with the target population. For example, the Massachusetts Juvenile Justice PFS Initiative tracks five outcomes. Three of those outcomes are tied to payments (number of jail bed-days avoided, job readiness, and increases in employment), while two additional outcomes (GED/high school completion and college enrollment rate) deepen the project’s understanding of the intervention’s impact on the participants’ lives.

In addition to tracking outcomes data, evaluations can assess a host of process indicators, or data on the process of service delivery itself. Such information—if tracked regularly throughout the contract period—would enable project partners to take timely corrective action and, after project completion, to determine what needs to be done to improve future service delivery quality.

For example, a service provider might implement a five-year PFS project that seeks to increase the number of disadvantaged, chronically homeless families placed in a permanent home for at least 12 months, with payments based on the number of families placed. The provider could also track the number of months each household waited until they were placed into such housing and tabulate these data to calculate the average time it took to achieve the permanent placements. Even though the wait time would have no bearing on the repayments made in the PFS project, such information could be useful in program planning and modification.

Specific questions that broader outcomes data collection can help answer include

  • Are success-payment outcomes affecting the other outcomes being tracked in the evaluation?
  • To what extent do success-payment outcomes appear related to other outcomes?  
  • Has the focus on the success-payment outcomes occurred at the expense of other quality-of-service outcomes important to service recipients?

While this additional data collection is useful, immediately and in the longer term, it may necessitate new data collection procedures, such as surveying individuals who have received services for ratings of additional service qualities. Or, as in the housing example above, the government or independent evaluator might tabulate wait times from administrative data. While these added procedures require special effort, seeking such additional information is good management practice in today’s recipient-centered public administration environment.

Have a Pay for Success question? Ask our experts here!

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research. Photo via Shutterstock