- Getting Started with Pay for Success
- Pay for Success Stakeholders
- Bringing Pay for Success to Your Community
- Current Pay for Success Projects
- Putting Together A Pay for Success Project
- Evidence, Evaluation, and Pay for Success
Getting Started with Pay for Success
What is pay for success?
Pay for success, or PFS, is an innovative financing mechanism that shifts the risk of implementing a public project from a traditional funder (usually a government) to a new private or nonprofit funder. The PFS project typically scales an evidence-based program to improve outcomes for a vulnerable population. If an independent evaluation shows that the intervention achieved agreed-upon outcomes, then the traditional funder repays the new funder’s investment with interest.By prioritizing evidence, outcomes, performance management, and the strategic deployment of resources, PFS could improve delivery of social services to vulnerable populations, yielding positive benefits to individuals, governments, and society at large.
What is the Pay for Success Initiative?
With support from the Laura and John Arnold Foundation, the Urban Institute launched the Pay for Success Initiative (PFSI) in 2015 to strengthen the role of evidence in PFS projects. Through this initiative, Urban is building tools to scale the PFS sector; designing, assisting, and assessing PFS projects; and sharing lessons learned with key stakeholders through communities of practice.
How do pay for success and social impact bonds differ?
Colloquially, "pay for success" and "social impact bonds" are used interchangeably to describe innovative financing of social or environmental interventions. However, PFS is a broader term that includes other methods of paying for outcomes such as performance-based contracting and results-based financing. In the United States, the phrase “pay for success” is preferred over “social impact bond” because the transactions do not perform like a traditional bond structure.
What are the benefits of pay for success?
Pay for success has several potential benefits, including:shifting the risks of innovating or scaling social services from government to private or nonprofit investors while raising new capital;advancing collaboration across government agencies and new partnerships between government and diverse stakeholders to achieve effective service delivery;focusing all stakeholders on outcomes rather than outputs or activities;encouraging the adoption of evidence-based solutions to serve the public interest;funding preventive programs, which maximize public benefits relative to costs over a longer time period;providing a framework for assessing local needs, capacities, and opportunities through data collection and mapping, and public-spending strategy conversations;generating new evidence on what programs are effective; andadvancing longer-term systems changes.
What are the challenges of pay for success?
Developing an outcomes-based contract with a clear target population, specified intervention, data collection methods, and robust evaluation is difficult. Although this may represent best practice for government service delivery, it is not easy.Most PFS projects thus far have required a substantial investment of time and energy from all partners. Projects typically take a year or more to launch. They require extensive data analysis, intervention and evaluation design, and contract negotiation. That said, this PFS structuring work can have benefits that reach far beyond the project, such as seeding systems changes within government. Attracting private investors may be a challenge given the low risk-adjusted returns that these projects often offer. However, there are a number of tools and options that projects may use to offset some of this investor risk, including encouraging philanthropic investors to serve as junior lenders or provide other credit enhancements.
What risks does pay for success present?
There are several commonly cited risks of PFS. Most can be eliminated or offset with proper planning. Governments and other stakeholders face financial, political, and reputational risks by engaging in PFS projects, underscoring the importance of a carefully managed and transparent process.A bigger-picture risk of the model itself is that PFS projects may measure the “wrong” outcomes—that is, outcomes that are easily measured and quantifiable instead of outcomes that could be more beneficial to society but are harder to identify or take years to manifest. Government and other stakeholders have lessened this risk by carefully selecting several evidence-based outcomes to measure.
How is pay for success different from traditional financing for public services?
Today, governments pay up front for publicly-funded goods and services through their traditional budget processes, regardless of whether the programs and products they’re funding will have a positive impact. Furthermore, government programs typically count and report inputs and outputs – for example, dollars spent and number of students taught– but rarely evaluate specific target outcomes, such as an increase in the graduation rate. Even rarer, few try to isolate and measure the impact of the program on those outcomes.In other words, today, governments are spending money on programs without knowing if they’re effective.In a PFS project, a third party pays for the program up-front, instead of the government. The project measures agreed-upon outcomes and estimates the intervention’s impact on those outcomes. Only if the project meets those outcome targets does the government pay.
How do PFS projects help vulnerable people?
The programs funded by PFS can improve lives of vulnerable people by supporting and scaling critical, evidence-based services, including permanent supportive housing for people who have experienced chronic homelessness, early childhood education for children in underserved neighborhoods, and job training for formerly incarcerated people. The focus on paying for outcomes allows government to pay only if the outcome of those services shows a benefit to the individuals served and/or society.
Pay for Success Stakeholders
Who are the main stakeholders in a PFS project, and what are their roles?
Six actors are essential to a PFS project:Governments identify social or environmental problems, select interventions, and pay if the program successfully achieves agreed outcomes.Private and philanthropic funders provide up-front financial support for the project and receive a return if the project is successful. Financial intermediaries structure the project and raise the up-front funding. A single organization or multiple organizations with different responsibilities may fill this role.Knowledge intermediaries identify and recommend high-performing programs or interventions, help structure the project, provide course-correction during implementation, and, with the independent evaluator, conduct ongoing research on impact.Service providers implement the evidence-based social program in the PFS project.Independent evaluators determine if the intervention has met desired outcomes.
Why are governments interested in PFS projects?
Today, governments bear all the risk of programs intended to serve the public interest. PFS transfers the risk of program innovation to private and nonprofit funders. PFS ensures that the government pays only for programs that improve social outcomes. There are also added benefits derived from the collaborative structure of PFS, whereby stakeholders are encouraged to create and sustain local buy-in to improve outcomes. Broadly, PFS can help advance evidence-based policymaking and strategic thinking within government systems.
Who invests in PFS projects?
PFS projects are impact investments and attract a variety of investors interested in potentially achieving both financial and nonfinancial (societal good) returns. These investors typically fall into two groups: private investors and nonprofits/foundations.Private investors, such as Bank of America Merrill Lynch and Goldman Sachs, are motivated by a potential return on their investment and the chance to enable verifiable social and/or environmental benefits. These benefits—the target outcomes of PFS projects—are increasingly valued by banks and the high-net-worth individuals who do business with them.Despite this interest, many PFS projects carry risks that are insufficiently offset by the modest returns on offer. As a result, private investors use a range of tools and approaches to mitigate or manage risk, including blending their capital with that of foundations.Foundations and other philanthropic funders—including the Kresge Foundation, Laura and John Arnold Foundation, and United Way—have played an important catalytic role in launching PFS projects by offering guarantees to entice private investors, providing grants for feasibility or pilot programs, or investing as senior or junior lenders. Foundations have been attracted to PFS because of its emphasis on results, use of evidence, potential for systems-change, multi-stakeholder approach, and potential for capital recycling.
What are financial intermediaries?
Financial intermediaries are third parties that assist in a variety of tasks during project development and implementation, including designing and structuring the project, securing investors, selecting the service provider, coordinating stakeholders, negotiating the contract, and providing oversight and project management. They are also known as transaction coordinators. Most projects have retained one. Their roles and responsibilities vary by project; some projects have even enlisted their help with project scoping and feasibility work.
What are knowledge intermediaries?
Knowledge intermediaries help PFS partners understand the research behind their target population and/or chosen intervention and translate that evidence into rigorous and objective projects. Knowledge intermediaries can be parties outside the project or serve another role, such as financial intermediary or evaluator, with the same project. The Pay for Success Initiative at the Urban Institute is a knowledge intermediary.
What are evaluators?
Evaluators are third-party people and/or organizations that objectively determine if the project being implemented successfully achieved its agreed-upon outcomes. In PFS projects, evaluators should be brought in as early as possible to assess the feasibility of the intervention including the likelihood of successful implementation and the potential for rigorous evaluation. Evaluators often work with funders and implementation partners to design the project’s evaluation. Finally, they carry out the evaluation to ensure the project is implemented with fidelity and measured as rigorously as possible, so that outcomes payments – or the determination that outcomes were not met and no payments are required – are legitimate.The Urban Institute serves as an evaluator for two active pay for success projects in the United States: the Massachusetts project on juvenile justice and reentry and the Denver project scaling permanent supportive housing to tackle chronic homelessness.
What is the difference between outcome evaluation and ongoing monitoring and evaluation?
The project evaluator plays an important role in ongoing monitoring and evaluation, as well as the outcome evaluation in a PFS project. The outcome evaluation is the final determination at the conclusion of a PFS project that the project achieved (or didn’t achieve) success as measured by agreed-upon outcome targets. Ongoing monitoring and evaluation is a continuous process in which the PFS parties assess program data in real time to track progress and inform midcourse corrections. Ongoing monitoring and evaluation ensures that the program is being implemented as intended and that adjustments are made as needed to improve the program’s likelihood of success. Ongoing monitoring is used to determine whether the funder receives interim payments.
What characteristics and experience should a PFS service provider have?
Service providers are important stakeholders and should be involved at key stages of PFS project design, negotiation, implementation, and evaluation. Strong providers should ideally exhibit the following characteristics:Subject-matter expertise in the topic area (and especially with the target population)Leadership commitment and interest in the PFS projectExperience implementing the specific interventionA track record of successfully delivering and documenting positive outcomesWillingness to engage in rigorous evaluation of its programExperience with performance-based government grants or contractsMore on this can be found in the PFS Project Assessment Tool.
Bringing Pay for Success to Your Community
What do you need to design a strong PFS project?
PFS can help communities address pressing social issues through collaboration, effective service delivery, and data collection and measurement. There are number of factors to consider when planning for a PFS project, such as the capacity of existing data systems and the strength of the program’s evidence base, among others detailed more fully in the Project Assessment Tool. Before deciding to pursue PFS, key considerations should include:the local context (i.e. scope & magnitude) of the problem that the project will address;the service provider’s demonstrated ability to successfully implement the social program;whether the proposed program is an effective solution for the identified problem;engagement and enthusiasm from public system partners, including government and political leaders;project alignment with local context and the unique cultural, societal, and geographic factors that might affect the program’s effectiveness, and performance and outcome tracking that estimates the costs of current services as well as the costs and potential benefits of implementing the PFS project.If you have specific questions about the enabling factors needed to implement a PFS project in your jurisdiction, ask an expert.
Why is it important for governments to create and sustain local buy-in?
Local buy-in is critical to the success of any pay for success project. Local stakeholders have an important role in informing and driving the conversation on key elements of project design, including issue identification and project scoping. Creating and sustaining local buy-in is important because it:helps projects get off the ground by providing demand and political support;enables the early identification and resolution of project challenges, strengthening the project's overall design;helps promote the PFS model's broader benefits, including the adoption of evidence-based frameworks for decision-making and strategic use of data;builds a constituency of support if the intervention is successful, which encourages sustainability, andcreates networks of local issue advocates that can outlast the project itself.
Current Pay for Success Projects
How many PFS projects are there?
There are 14 current PFS projects and one concluded project in the United States and many more projects abroad, primarily in the United Kingdom.
What are the issues that current PFS projects are working to address?
PFS projects explore a wide range of issue areas. In the US, the launched PFS projects have covered the following topics:Juvenile justiceRecidivismSupportive housing / homelessnessEarly childhood educationHome visitation/ pre-natal careGreen infrastructure
Where can I learn about existing PFS projects and outcomes?
The Urban Institute has produced a number of papers and blog posts describing PFS projects, their potential, and their limitations, and maintains a database of project fact sheets. In addition, the Nonprofit Finance Fund continually updates information about PFS projects and activities. The Harvard Kennedy School Government Performance Lab also offers various resources for potential PFS partners.
Putting Together A Pay for Success Project
What are the main steps of a PFS project?
After the strategic planning process is complete, there are six main steps in a PFS project:Problem identification: Analysis of inefficiencies and challenges the PFS intervention will address.Stakeholder engagement: Identifying and selecting all stakeholders necessary for successful project design and implementation.Examine the evidence base: Assess existing evidence on the identified problem and conduct data collection and analysis to determine the project scope and target population.Project design: Develop the PFS project, which includes selecting an intervention with a strong evidence base, modifying that intervention as appropriate for the local context and population, and selecting outcomes to measure performance.Implementation: Implement the intervention, adhering as close as possible to practices that the evidence suggests are effective.Evaluation: Assessing the intervention’s performance against pre-identified outcomes.
How long do PFS projects typically last?
In general, PFS projects should last long enough to deliver and measure program impact and potentially realize cost savings. Most existing projects have established a three- to five-year term for implementing the program, though the timeline for evaluation may exceed that window. The program should employ a robust monitoring and evaluation framework with mutually agreed-upon outcomes that are appropriate for the time frame selected.
What types of programs may be poor candidates for PFS projects?
Programs that are less-than-ideal candidates for a PFS project include the following:new programs that have not yet been rigorously evaluated, do not have a clear theory of change, and/or do not have access to data for a strong evaluationcore government activities (e.g., public K-12 education) that cannot be transferred to a nonprofit or private partnerprograms that can easily reach scale through conventional financing mechanisms, such as programs with clear benefits and costs that are self-contained and can be entirely funded by one government agency
What should stakeholders consider when pricing a PFS agreement?
Parties should consider the following factors when building a financial model to price a PFS deal:current cost of services to the governmentoperational and capital costs associated with implementation of the programtransaction costs (e.g. legal project management costs)cost savings as a result of the proposed program and the ability to quantify those cost savings financially (“cashability”)risk, which is broadly a function of general factors (e.g., robustness of the evidence base) and context-specific factors (e.g., unique aspects of the target population, features and capacity of the implementing agency, composition and ambition of the outcomes)evaluation costs
What information should a PFS contract include?
Structuring clear and transparent PFS contracts is vital to ensure that all parties are aware of their roles and responsibilities, agree on what a successful project looks like, and how the project should be measured. Depending on the PFS project, there may be separate contracts between involved parties. Contractual agreements should clearly identify the following:social problem, intervention design, target population, enrollment process, and evaluation design;financial information (e.g., costing, budget) and source of the outcome payment;time frame for implementation;implementation commitments, including authorized budget and planned activities by implementing agency or party;handling and selection of subcontractors;monitoring and evaluation framework that clearly details the process and procedures for measuring and evaluating outcomes;termination criteria; andthe outline of necessary data and where to find them, or how data should be collected.
What is the "wrong pockets problem"?
The wrong pockets problem describes a situation in which the entity that bears the cost of implementing a practice or program does not receive the primary benefit. For example, an effective intervention for high school students at risk of dropping out might ultimately reduce the number of students who end up involved in the criminal justice system. The criminal justice agencies will reap the benefits of this reduction in crime - as will the community - but the entire cost of the program may be borne by the education system. If the costs of the program outweigh the benefits for that entity, it is unlikely to fund it, even if the net benefits for the government (and society) are strongly positive. PFS can help solve the wrong pockets problem by involving a number of stakeholders as outcome payors.
What’s the track record of the PFS model?
Determining if interventions are successful takes time, so only a few PFS projects have delivered results so far. However, the model itself can “work” regardless of whether the intervention meets its outcome targets. If a project does hit its outcome targets, as is suggested in the initial results from the Utah High-Quality Preschool program, the project is a success for government because it has led to beneficial social outcomes.However, even when the intervention misses its outcome targets—which occurred in the NYC ABLE Project for Incarcerated Youth—the project may still be valuable for government because the government will not lose money and will gain valuable insight on effective project design and what works with the target population.In addition, HUD and DOJ are currently examining how multiple demonstration sites evaluate expanding supportive housing using the PFS model. Initial results from these sites have offered lessons on the importance of data and the stakeholder buy-in. It is still too early to tell whether the PFS model, including the entire project design and implementation process, advances broader systems changes that drive more evidence-based policymaking processes as hoped, but anecdotal evidence suggests this may be occurring.
What happens after a PFS project concludes?
With PFS projects in the US still underway, clear evidence of the long-term impacts of the PFS-funded programs on the people they serve is not yet available. However, there are many positive impacts that have emerged during PFS project implementation. PFS can create partnerships that may continue beyond the life of the project. Establishing data sharing agreements, developing special interest coalitions, and other engagements, will facilitate further collaboration among stakeholders after project conclusion.
Evidence, Evaluation, and Pay for Success
What is evidence?
Evidence is the available body of facts or information indicating whether a belief or proposition is true or valid. In social science, that available body of facts (the evidence base) is built through testing and evaluating hypotheses that link a specific intervention with target outcomes.
How does PFS use evidence?
Evidence is central to PFS. It is used to shape strong PFS projects, which in turn generate evidence to inform better public policy. There are three important ways evidence helps drive PFS project development: guiding project scoping, informing program selection, and influencing terms of the deal. The ability to interpret and assess the quality of existing evidence is important, and is one of the roles an evaluator can provide if brought on early.
What is meta-analysis?
Studies of program or intervention effectiveness may not always agree on whether the program is successful or, if so, how effective it is. To overcome this challenge, a statistical method called meta-analysis enables a single estimate of effect size from multiple studies with potentially conflicting results. Meta-analyses do not just average the studies but also consider other factors such as the quality of each study and sample size.
How important is access to quality data in PFS?
Essential. Having reliable, high-quality, timely, and comprehensive data is necessary to determine the eligibility criteria for service recipients, successfully implement the program, and measure results. Creating integrated data systems (IDS), which link data across different sources, can help track and measure the impact of current PFS projects.
How is the success of a PFS project evaluated?
PFS projects are subjected to an evaluation, in which the independent evaluator determines if the project has achieved the agreed-upon outcomes. These evaluations determine the impact of an intervention and whether the government should pay the investors; they also build the evidence base on effective interventions for specific social problems.
What are common evaluation methodologies?
Stakeholders should carefully select the evaluation design their project will use to determine program impact. Some of the most common methodological approaches include randomized control trials (RCTs), regression discontinuity design, difference-in-differences comparisons, and historical baselines. These approaches aim to compare the outcomes of those receiving the treatment against what would have happened otherwise (counterfactual). This counterfactual can be created a variety of ways—this is the primary distinction between the different evaluation design options. Because RCTs use random assignment and are therefore a true experimental research design, they are often considered the “gold-standard” methodology and are typically the most effective and efficient way to evaluate a program or practice. Although some projects may opt not to use an RCT for a variety of reasons, evaluation conversations should start with them, since they are the most rigorous design and provide the best estimates of causal impact.
What is a comparison group?
The goal of a high-quality evaluation is to estimate the effect of the treatment on those receiving it. To estimate effects well, there needs to be something to compare the treatment group against. The comparison group approximates the counterfactual: how the treatment group would have fared without the intervention. The more similar that comparison group is to the treatment group, the stronger the evaluation can be. Comparison groups are selected in RCTs through random assignment, which is the closest applied way to replicate the counterfactual.
What are the most likely barriers to high quality evaluation?
There are many considerations for designing and implementing high quality evaluations. Some of the common challenges that arise include:A lack of data that accurately describe the comparison. This might be because data systems do not communicate or because high-quality data are only collected for program participants (not the control group as well).Selecting too many outcomes. Evaluations should choose a limited number of measurable outcomes with clear payment triggers.Incomplete or inefficient data systems. Evaluations require data systems that are reliable, high-quality, timely, and comprehensive.Costs and complexity. Project stakeholders may perceive that a project design is overly complex or expensive. Without understanding the merits and limitations of different design options, decisions may be driven by incomplete information.
How do partners set outcome targets for a PFS project?
One of the most important characteristics of PFS is its grounding in performance and articulation of specific measurable outcomes that define success. Intermediate metrics should allow for midcourse corrections, and outcome targets should be ambitious yet achievable. Targets should be chosen to reflect meaningful outcome impacts for beneficiaries and, if possible, correspond to savings for the government. Therefore, the selection of outcome targets should flow from initial discussions about the government’s desired objective. Ensuring that all parties in the PFS agreement understand and agree with how success is being measured is essential to avoid confusion, complication, and disputes at implementation conclusion and evaluation.
What is a randomized control trial, or RCT?
A randomized controlled trial (RCT), or experiment, randomly assigns eligible people to a treatment group, which receives services, or to a control group, which does not. As a result, the differences should reflect only the direct effect of the program. The RCT design accounts for all other competing explanations. This method produces causal results with the highest confidence and clarity.
What is a quasi-experimental evaluation method?
Quasi-experimental designs are similar to RCTs in that they compare the outcomes of a group that received an intervention against those of a group that did not receive the intervention. However, quasi-experimental designs create their comparison group by matching individuals with the treated individuals on a number of factors, instead of through randomization. These non-randomized control groups pose bias risks and are typically considered less rigorous than RCTs. Designs using difference-in-differences, paired testing, and regression discontinuity are examples of quasi-experimental evaluation designs.
What is regression discontinuity design?
In this quasi-experimental evaluation design, outcomes are compared from two groups: those who score just above, and those who score just below, the eligibility threshold (e.g. test scores or income level). This approach is based on the judgment that the difference between one scoring 51 and another scoring 49 (with a threshold of 50 out of 100) is negligible. This approach is suitable only for evaluating programs with quantitative eligibility thresholds.
What is a difference-in-difference comparison?
In this quasi-experimental evaluation design, two populations are examined with similar characteristics (including risk factors) except one participates in the program and the other does not. This approach then compares the difference in outcomes experienced by each group over the course of program delivery.
What is the historical baseline approach?
In this quasi-experimental evaluation design, outcomes experienced by the target population receiving the program treatment are compared to outcomes for a similar population in the past. Without a control or direct comparison group, it’s difficult to identify with much confidence whether the outcomes would have occurred in the absence of the program. When the baseline is used for a specific group, this is a pre-post analysis or benchmark design.
What is a rate card?
Rate cards are non-experimental designs that present a menu of outcomes priced by the government beforehand to help clearly identify priorities. Under a rate card scheme, when an outcome is achieved in a project, it triggers success payments by the government at that agreed-upon rate. However, there are several limitations to the model, mainly the lack of rigorous evaluation.