Exploring criticisms of pay for success

This is the first of a two-part blog series considering criticisms of pay for success (PFS). This series builds on an opinion piece written by our Initiative’s co-director, Justin Milner, which addressed four of the most common criticisms of PFS: that PFS projects ‘privatize’ public services, that governments should just pay for the program outright, that only funders benefit from the model, and that projects have high transaction costs. This post considers three additional criticisms and an accompanying blog offers some concluding thoughts on the risks and challenges of PFS. The second post offers some concluding thoughts on the risks and advantages of PFS. 

Though some have billed it as a breakthrough moment for the public sector and others have noted its potential to shift governments towards more evidence-based policymaking and a “culture of outcomes”, pay for success (PFS) is not without its detractors.

Critics have raised concerns about the pay for success model on both practical and theoretical bases. These concerns range from unsubstantiated claims to serious critiques meriting closer consideration. This blog will consider three common concerns about PFS—and why and how they are surmountable.

1. Early termination clauses create risk for governments. 

A benefit of PFS is that it shifts the financial risk for a new social program from the government to third party funder(s). However, some critics suggest that PFS actually saddles governments with additional risk. This critique, raised in, among other places, a 2013 report prepared for Maryland, is based on a hypothetical scenario: A government pays  to set up a PFS project, but because preliminary evaluation results suggest the project is ineffective, the funder exercises an early termination clause and ends the project. As a result, the government, despite having contributed funds to launch the project, is exposed to early termination risk.

Early termination clauses in PFS contracts, where they exist, allow funders to cut losses if preliminary evaluations suggest a program is ineffective. These clauses help funders mitigate project non-performance risk—which would cause the funder to lose the entire investment—and allow all parties to stop participating in a project that isn’t working and divert resources to more effective uses. Governments can avoid the “risk” that an funder will exercise an early termination clause by excluding such a clause from the project contract, or by setting strict conditions under which an funder can exit early. Further, while governments do expend resources to set up a project, so do funders, and as such, they too have a disincentive to terminate a project early without clear reason.

Nonetheless, it’s true that governments face financial, political, and reputational risk by participating in PFS projects, particularly if the project is poorly designed or communicated. PFS should not be sold as “risk free” for government—risks are inherent in trying something new, regardless of the financing mechanism. In PFS, at least, the bulk of the financial risk is shifted to the funder. Even though the government may absorb a portion of the costs of an unsuccessful program, it is still in their best interest to end a project that is not working. 

2. The risk-reward trade-off isn't worth it or sustainable for private funders. 

While many PFS projects have had a mix of private and philanthropic funders, private interest may be unsustainable due to insufficient financial returns relative to risk. Because PFS is a new tool, it has attracted considerable public and political attention. As a result, funders may be willing to accept relatively low returns in exchange for public relations wins with shareholders, consumers, and policymakers. Additionally, philanthropies have played a large role in helping private funders mitigate risks. Philanthropies’ involvement—through blended capital and other means—is often seen as “catalytic” and without it, private funders may avoid investing in PFS projects altogether.

For this reason, some suggest that the future of PFS will rely increasingly on philanthropic funders, rather than for-profit ones. But there are also reasons to be skeptical of this concern.

First, as PFS grows, the field will likely develop more and better tools to help appraise, manage, and mitigate risk. This will enable parties to enter project negotiations better informed of what an appropriate return might be relative to the risk and to identify lower-cost solutions to reduce or offset that risk. Further, the field may discover a “sweet spot” of project risk high enough to encourage governments to pursue PFS but low enough for them to afford potential outcome payments. Interventions with a good evidence base but without proven successful application in the specific location might be good candidates for this target risk level.

Second, while it’s unlikely governments will be willing or able to offer higher returns in the near future, private funders, particularly impact funders, may accept the fact that PFS projects yield less than many other investments with similar risk, in favor of a double bottom line—the potential to reap both social and economic returns.

Third, banks are just beginning to explore the potential of PFS projects to satisfy Community Reinvestment Act (CRA) requirements. If PFS projects are considered a valid investment to meet CRA, this would increase banks’ tolerance for lower returns through PFS. , While this could raise concern if it diverts CRA funds from other worthy investments, an increase in CRA-eligible investments presents an opportunity that would enable flexibility and innovation in this space.  

3. PFS projects measure the wrong outcomes. 

One of the strongest critiques of PFS is the idea that PFS encourages measurement of the wrong outcomes—that is, outcomes that are easily measured and quantifiable instead of outcomes that are harder to identify or take years to manifest.

This concern channels the warning of Campbell’s Law: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

This is certainly a risk and, although not specific to PFS, is one that PFS is particularly exposed to because it emphasizes measurable outcomes. However, just because governments run the risk of measuring the wrong outcomes doesn’t mean the practice of measuring outcomes is itself bad. Indeed, it is a substantial positive step for governments to start measuring the outcomes of their spending—something that is rarely, if ever, done in most jurisdictions.

Government and other stakeholders need to be careful about how they decide to measure success, and select outcomes in a way that is transparent, evidence-based, and informed by subject-matter and evaluation experts. Another way some governments have lessened the potentially perverse impact of measuring the wrong outcomes is by evaluating the project’s success using several carefully selected outcomes, even if the outcome payments are determined only using one or two.

Have a Pay for Success question? Ask our experts here! 

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research. Photo via Shutterstock.