In April, the Forum for Youth Investment and the Urban Institute brought together policymakers and practitioners from across levels of government and the non-profit sector to discuss Using Evidence for Improvement in the Foundations for Evidence-Based Policymaking Act. The event featured a panel of speakers from federal, state and local agencies to share how they have used evidence to improve programs, with closing remarks from the White House Office of Management and Budget (OMB) on what agencies should consider as they begin to implement the Foundations for Evidence-Based Policymaking Act (Evidence Act).
Panelists included Jean-Marie Callan from the New York City Mayor’s Office of Economic Opportunity (NYC Opportunity), Peter Nelson from ServeMinnesota – a statewide administrator of Americorps funding, and the U.S. Department of Housing and Urban Development’s (HUD) Barry Steffen. The panel was moderated by the Forum for Youth Investment’s Executive Vice President, Thaddeus Ferber. Diana Epstein from OMB provided closing remarks.
Panelists, whose agencies the Forum for Youth Investment profiles in a series of case studies, expounded on the process behind harnessing evidence to improve programs:
- NYC Opportunity added an internship component to their Young Adult Literacy program (an education program focused on young adults with a fourth to eighth grade reading level who are hoping to join a GED-type program in the future) after research showed that the program struggled to retain participants despite the program’s positive outcomes.
- ServeMinnesota added a weekly monitoring component to their K-3 literacy tutoring program Reading Corps after research showed that some students regressed in literacy skills after leaving the tutoring program and returning to a normal classroom schedule.
- HUD improved their housing choice voucher program by using data to guide voucher users towards higher opportunity neighborhoods, which are linked with better outcomes particularly for young children.
The panelists explored how research led them to consider program changes, how they implemented change, and how they examined whether or not they successfully improved outcomes for children and youth. They highlighted the types of evidence and diverse approaches they pursued to best understand their programs’ effects, including:
- ServeMinnesota’s crucial investment in data infrastructure to track information on individual students and schools, ensuring they could understand for which populations and under which contexts Reading Corps program worked best.
- How NYC Opportunity combined qualitative implementation studies to show which sites are utilizing best practices with a longer impact study to discover which student populations benefited the most. Young people entering the program with a fourth to sixth grade reading level demonstrated the most improvement, enabling NYC Opportunity to further target the program and improve site implementation.
- How HUD used random assignment methodologies to simplify understanding of complex housing interventions and looked across data sets at different agencies to explore broader outcomes measures for children and families.
In closing, panelists shared advice for federal policymakers on using evidence for improvement:
- Peter Nelson emphasized having a well-articulated theory of change in order to get at multiple key outputs and outcomes the data sources needed to understand them.
- Jean-Marie Callan said organizations should prioritize flexibility so that researchers can meet programs where they are.
- Barry Steffan stressed that the priorities and goals of the Evidence Act can support meaningful improvement and not just ‘check the box’ exercises.
OMB’s Evidence Team Lead Diana Epstein provided closing remarks on the implementation of the Evidence Act. The Act implements about half of the Commission on Evidence-Based Policymaking’s 2017 recommendations to Congress, addressing evaluation, open data, and statistical and privacy requirements.
As OMB focuses on putting out guidance for federal agencies, Diana Epstein stressed the importance of agencies working across offices and functions to implement the act. She emphasized that no single person can do this work alone. While the Act creates new positions, such chief evaluation officers and chief data officers, these new leaders will need to work together across silos. She also emphasized how the learning agenda processes within the Act can help agencies partner with external groups as well as provide opportunities to collaborate internally across different offices.