Urban Institute
Research Analyst

Randomization in practice

April 11, 2018 - 11:33am

The contractual nature of evaluation in pay for success (PFS) means that partners must agree on the design.  Sites looking to develop the evaluations that can determine the causal impact of the intervention may choose to use a randomized controlled trial (RCT) design. For some service providers, RCTs can pose ethical challenges. For PFS partners considering an RCT, addressing evaluation questions early in project development can help evaluators design service provision to manage ethical concerns around randomization. Below, we’ve highlighted four strategies to implement randomization in practice.

1. Randomized referrals. In the Denver Social Impact Bond Program on chronic homelessness, individuals from the target population appear in the evaluators’ electronic system upon arrest. Individuals are then randomly assigned to either the treatment group, who are referred to the Denver SIB permanent supportive housing (PSH) project, or the control group, who are not referred. Project staff responsible for recruiting participants are only aware of who is in the treatment group, and so target street outreach to only those individuals. Since project staff do not know who is in the control group, individuals randomized to control just receive status quo services (see below). This allows service providers to avoid the unsettling situation of turning willing and eligible people away in person, and recruit only those people who have already been selected to receive services.

2. Cluster Randomization. In cluster randomization, groups, rather than individuals, are randomized. That is, randomization occurs at a higher level, such as the county level. For example, all counties in a state may be randomized to either provide eligible people with a new workforce development program, or not. In this case, the aggregate outcomes of people in counties in the treatment group—that is, delivering the new workforce development program—would be compared to the outcomes of people in counties who did not implement the program. This avoids having to deny individual people services.

3. Delayed service provision. PFS partners could mandate that program services be extended to people in the control group if the RCT shows that the intervention has improved outcomes for the treatment group. For example, let’s say that three years of data collection and analysis in an RCT demonstrates a significant difference in outcomes between the treatment and control group. The PFS contract could state that in that case, program services have been shown to be effective, and can then also be delivered to the control group. Delayed service provision may require additional funding, either through PFS channels or elsewhere.

4. Enhanced treatment. To avoid denying people all beneficial services, it is possible to design a RCT so that some get treatment while other receive enhanced treatment. For instance, the treatment group could receive housing in addition to wraparound services, while the control would receive just housing. The results of this RCT would indicate whether housing with services produced better outcomes than housing without services.

Keep in mind that people in the control group, particularly if they comprise a vulnerable population, can still access status-quo services—resources, amenities, and services already available in the community—regardless of the randomization strategy. This is a possibility when the local jurisdiction already has programming for the target population. In this scenario, the purpose of the PFS project would to implement a potentially more effective intervention, rather than trying to fill a gap in service provision. The purpose of an evaluation is to determine whether the intervention produces better outcomes than the services currently being delivered (if any). Even though they cannot receive program services, people in the control group can still connect to other potentially beneficial services.

Successfully implementing an RCT requires PFS stakeholders to ask the right questions, balance research aspirations against practical constraints, ensure proper randomization processes, and more. If PFS partners do choose to do an RCT, it is important for them to realize that there are ways to reap the benefits of random assignments while still assuaging the ethical concerns of partners.

Have a Pay for Success question? Ask our experts here!

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research. Photo via Shutterstock.