The role of data in pay for success and early childhood education

Collecting reliable and informative data is an important part of any pay for success (PFS) project. Data can inform whether a PFS project is feasible, who should be served, what outcomes are most appropriate, and how the evaluation should be carried out – and, at its conclusion, whether a project is successful in achieving its target outcomes.

Using Data to Inform Decisionmaking is part of the recently released Pay for Success Early Childhood Education Toolkit. It is one of seven toolkit products, produced in collaboration with Urban’s Pay for Success Initiative, Urban Institute experts, and external partners. This report provides a roadmap for stakeholders on best practices for collecting and using data when designing and launching PFS. It is targeted toward states and localities considering PFS projects in early childhood education (ECE) specifically.   

How can data be used in PFS projects?

Data are critical tools for all stakeholders involved in a PFS project, and the quality and availability of data affect decisions in every step of project development:

1. Project feasibility: When assessing the feasibility of a PFS project, historical data can help investors understand past program performance, local demand for services and expected enrollment, and the demographics of the target population. This information gives stakeholders a better idea of the level of unmet need in the community and helps gauge whether pursuing a PFS project is appropriate.
2. Developing the project: Once stakeholders have decided that a PFS deal is feasible, they can use data to further solidify the logistics of the project. Using data from past evaluations, for example, can help inform the selection of a program model, identify staffing and other resource needs, and define repayment conditions, such as outcomes and outcomes pricing.
3. Informing the evaluation design: Data also inform stakeholders on decisions such as whether the program can support randomization, the options for a comparison group, and the ideal sample size, which is critical for the evaluation design and PFS project as a whole.
4. Monitoring implementation: After the project has been launched, data need to be collected on how the intervention is being implemented, if it is being implemented as intended, and what factors have facilitated and hindered program implementation. This information can help explain why a program is not achieving its desired outcomes.
5. Evaluating outcomes: Outcome data must be reliable, valid, and analyzed appropriately to determine the impact of the program and whether it achieved its desired outcomes.

Given the importance of data at each step in this process, having a clear data plan is critical. When beginning a PFS project, stakeholders should ideally outline how data will be collected, integrated, and analyzed over the course of the project, including:

• Asking what the program is trying to achieve. This research question will inform data needs. Stakeholders must determine if the data they need exist and, if so, what kind of shape they are in, how reliable they are, and the steps required to obtain them. PFS partners need to consider both the availability of existing data as well as the need to collect new data to measure desired outcomes.
• Identifying data sources. Once stakeholders know what data they need, they can pinpoint data sources. There are many potential sources for ECE data, including education, health and human services, and housing databases. Stakeholders should map out existing data systems, figure out what they contain, and determine whether and how they connect. Relevant state agencies and partners can offer guidance on available data sources during the project feasibility stage.
• Establishing data-sharing agreements if necessary. If the project requires that stakeholders share data across systems, they must create data-sharing agreements that comply with confidentiality and consent procedures. Being reasonable about how many data requests are necessary is important, since asking for too many may strain relationships with agency staff.
• Being aware of possible challenges. Stakeholders are likely to confront some data challenges and should be proactive in anticipating them and developing potential solutions. For example, data systems may change over time, datasets may have missing or unavailable data, or children may move from the community or withdraw from the participating school district.
• Ensuring that the right data infrastructure is in place.  Various products on the market offer technology tools and solutions to support data storage, data integration, and data analytics. PFS stakeholders should consider which kind of technology will best capture the complexity of their deal and best serve the project’s goals. 

Using data to inform decisionmaking throughout the course of a PFS project helps ensure project success, but it is only one part of the equation. Additional toolkit reports will provide information on the other critical aspects of PFS projects, including outcomes measurement and pricing, program funding and financing, program implementation, and evaluation design.

As an organization, the Urban Institute does not take positions on issues. Scholars are independent and empowered to share their evidence-based views and recommendations shaped by research.