Urban Institute
Policy Program Manager

Future of PFS: Center for Employment Opportunities and performance management in PFS

A Q & A with Christine Kidd, Director of Program Innovation, and Mary Bedeau, Deputy Executive Director at CEO-NYC
February 1, 2018 - 1:34pm

At the 2017 National Symposium on the Future of Pay for Success, the “Managing for Performance” panel brought together researchers and practitioners to discuss the role of performance management in the Pay for Success context. One of the panelists was Christine Kidd, Director of Program Innovation for the Center for Employment Opportunities (CEO). CEO is dedicated to providing immediate, effective, and comprehensive employment services to men and women with recent criminal convictions. Founded and headquartered in New York City, CEO now offers services in 18 cities nationwide. CEO is the service provider in the first state-level PFS project in the country and has pursued other performance-based contracting models as well.

At the event, Christine spoke to the challenges and opportunities of PFS project and performance management for CEO’s ongoing pay for success projects. We followed up with Christine and Mary Bedeau, the Deputy Executive Director of CEO-NYC, to learn more.

PFSI: Can you tell us about CEO’s overall approach to performance management?

CEO: At CEO, we set annual organization-wide and site-specific goals related to key performance metrics. They’re tracked closely in our Salesforce platform and reviewed regularly by staff and leadership.

Leading indicators include our enrollment numbers, the percent of participants who complete orientation, and attendance rates at our job sites. These correlate with the significant milestone of job placement; we set ambitious goals about the number of job placements we make and the average wages of those jobs. Once participants find work, we track job retention with a special focus on 90, 180, and 365 days of job retention. We break out our data between young adults and their older counterparts to keep ourselves accountable to our young participants who often struggle in the labor force.

With the support of Fund for Shared Insight, CEO has also begun to incorporate participant feedback into our performance management practice. We collect feedback through text message surveys, focus groups, one-on-one conversations, and in-office tablets where participants can remain anonymous. These feedback channels integrate with Salesforce so we can look at organization-wide and site-specific performance on everything from response rates to the content of participants’ feedback, including CEO’s Net Promoter Score.

One metric that we cannot track in real-time is recidivism rate. We’ve been able to learn more about our effect on recidivism in PFS projects, but we struggle to get regular access to administrative data to track our performance in this area. Only a few of our criminal justice partners will share it in aggregate, and without consistent case-level information, it’s difficult to integrate it into performance management.

Our efforts around performance management have been pretty distinct from our evaluation work. In the coming years, we’re hoping to integrate the two types of work by using performance management data as a source of ongoing learning, and information from evaluations to inform performance management practice.

PFSI: How has performance management worked differently in the context of your PFS project?

CEO: It has differed in several regards:

  • Tracking new metrics. For a variety of reasons, the New York State project required us to track different metrics than standard practice. One example is the long-term employment measure. Typically, CEO tracks participants’ success in staying connected to a job for one year after they begin. In PFS, success is defined as connection to the workforce in the fourth quarter after an individual was released from prison. For example, if an individual was released from prison in the fourth quarter of 2013, success is defined as employment for at least one day four quarters later (between October 1 - December 31, 2014), regardless of when they enrolled in CEO or got a job. This required changes in our data system and program administration.
  • Focus on the front door. PFS is often held up as a way to get government and providers to shift away from a focus on inputs towards a focus on outcomes and impact. While long-term impact was the driving force behind the project, day-to-day management focused heavily on meeting enrollment targets. Normally, CEO’s performance management tracks progress throughout the program model, but the focus on enrollment numbers in PFS meant that much of our management energy shifted towards this metric. This was largely driven by the need to reach enrollment goals that were critical to investors and to sufficiently power the evaluation.
  • Responding to control group data. At CEO, we rarely have the chance to look at how a control group is doing in comparison to our participants. During PFS, we received monthly reports on control group performance. It was challenging, however, to figure out how to take appropriate action. We didn’t want to respond prematurely to control group outcomes given how preliminary the data was. At the same time, we didn’t want to ignore the data. Since it wasn’t part of our regular performance management practice at CEO, we didn’t have strategies on whether and how to respond.

PFSI: On the panel, you mentioned the importance of “collaborative performance management” with other partners in a PFS project. Can you say more about that?

CEO: One of the really positive takeaways for us from PFS was the way we were able to partner with New York State’s Parole agency. Under PFS, there was a shared sense of urgency around getting people referred to CEO.

This happened in a few ways:

  • Daily CEO presence in parole offices to work with parole officers on identifying PFS participants, and agreed-upon referral criteria to identify people who would benefit most from services
  • Weekly or bi-weekly calls among project partners about operations, research, and management, including tracking the number of eligible participants that were reaching CEO’s offices
  • Quarterly meetings between Parole staff and CEO staff involved in the project
  • Periodic governance calls where performance issues could be elevated

These new routines made a difference on many levels. Our Outreach Specialists were more integrated into Parole decision-making and processes than ever before, and the relationships developed there have helped to prevent problems and resolve them when they arise. The ongoing meetings and data sharing reinforced an understanding of which individuals were appropriate for CEO. As a service provider you often have to walk the line of being flexible and accepting the referrals that come your way, but also  really pushing to serve the people for whom you’re most effective. PFS brought us together around a shared understanding of who to serve and pushed Parole and CEO to rally around them.

PFSI: Are there other best practices for performance management you’d suggest to a service provider embarking on a PFS-funded project?

CEO: Service providers should recognize and accept that performance targets that are put in place are going to incentivize certain behaviors across the organization. Some of the targets can have unintended consequences that, with forethought, can be managed against. In a workforce project, for example, there might be a payment point related to job placement. While this metric is an important measure of success, if you push too hard on job placements you might have staff placing people in jobs before they’re ready just to hit their goal numbers. This might impact long-term retention. It can be helpful, with each metric, to ask, “How could this go wrong?” and devise a companion metric to protect against it. In the workforce example, you might add a goal around job retention to push for quality alongside quantity.

We would also advise providers to not underestimate the amount of performance management time that goes into PFS. You can see from the above meeting list alone that a lot of time was invested by all parties in managing performance. Providers should account for that in their PFS budget and personnel planning. 

This is the thirteenth blog in our Future of PFS blog series.

On June 22nd and 23rd, 2017, the Pay for Success Initiative hosted a National Symposium on the Future of Pay for Success in Washington, D.C. The invite-only Symposium brought together leaders from government, nonprofits, and organizations active in pay for success to consider the big questions facing the field, as well as highlight lessons for engaging in PFS efforts. More information on the Symposium can be found here.

Over the past several months, the Initiative released a series of blogs highlighting important conversations, themes, and questions that arose during the Symposium. To join the conversation, visit pfs.urban.org, @UrbanPFSI and #FutureofPFS on Twitter, and subscribe to our monthly newsletter.

Have a Pay for Success question? Ask our experts here!

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research. Photo via Shutterstock.