Advancing Evidence-Based Practices

While much attention is given to identifying and replicating effective “brand name” programs, it is also important to look inside the ‘black box’ of programs to identify the specific components that drive their success. These components are variously referred to as staff practices, active ingredients, and kernels. By identifying and promoting use of these components, researchers, policymakers and practitioners can better predict which types of programs are likely to succeed (including home-grown programs with budgets too small to warrant a rigorous evaluation), spur effective local innovations and adaptations, and improve the quality of existing programs and services.

Such approaches are being used in the afterschool field (through the Forum’s Weikart Center for Youth Program Quality), the juvenile justice field (through the Juvenile Justice Reform and Reinvestment Initiative), and the child and adolescent mental health field (through the work of Bruce Chorpita and others).

This webpage will serve as a repository for related publications and resources. Subscribe to our Evidence-Based Policymaking Newsletter to stay up to date on this and related topics.

 

 

Cross-Cutting

Better Evidence for Decision-Makers: Emerging Pathways from Existing Knowledge (2017)

Steve Cohen

The Center for the Study of Social Policy discusses how evidence can be used to improve results in an entire field or for a broadly defined target population.

 

Evidence-Based Kernels: Fundamental Units of Behavioral Influence (2008)

Dennis Embry & Anthony Biglan

This paper describes evidence-based kernels and suggests how kernels could augment existing prevention or treatment efforts, clarify the active ingredients in existing interventions, and efficiently develop new interventions.

 

Afterschool

Program Quality Assessment Handbook: Social and Emotional Learning (2017)

Hillaker, B., Akiva, T., Jones, M., Sutter, A., Wallace, L., Mcgovern, G., Harris, J., Peck, S. & Smith, C.

The Youth Program Quality Assessment (PQA), was designed to assess the quality of learning environments and to identify staff training needs.

 

Continuous Quality Improvement in Afterschool Settings: Impact Findings from the Youth Program Quality Intervention Study (2012)

Smith, C. Akiva, T., Sugar, S., Lo, Y. J., Frank, K. A., Peck, S. C., & Cortina, K. S.

This study, funded by the W.T. Grant Foundation, evaluates the Youth Program Quality Intervention (YPQI), a data-driven continuous improvement model for afterschool systems. Using a rigorous, experimental design, the research finds that the YPQI produces a cascade of positive effects, resulting in improved program quality at the point of service.

 

Moving the Needle on “Moving the Needle”: Next Stage Technical Guidance for Performance-Based Accountability Systems in the Expanded Learning Field with a Focus on Performance Levels for the Quality of Instruction Services (2013)

Charles Smith

This paper introduces the nomenclature of performance-based accountability systems (PBAS) to the expanded learning field, provides a policy case study for a countywide system in southern Florida and uses data from that system to explore the issue of quality thresholds. Findings suggest that (1) the PBAS framework defines critical concepts and improves our ability to describe existing quality improvement systems, (2) the Youth Program Quality Assessment (Youth PQA) can be used to produce a program rating of sufficient reliability for use in a PBAS, and (3) that the Palm Beach County PBAS design is an exemplar for expanded learning policies.

 

Full Findings from the Youth PQA Validation Study High/Scope Youth PQA Technical Report (2005)

Smith, C & Hohmann, C.

The Youth PQA Validation Study was a 4-year effort to develop and validate a tool to assess program quality in youth settings. In total, the validation study encompassed 59 organizations in Michigan and more than 300 Youth PQA observations and interviews conducted in programs serving 1,635 youth. In general, findings from the study demonstrate that the Youth PQA is a valid, reliable, and highly usable measure of youth program quality.

 

Preparing Youth to thrive: Promising Practices for Social and Emotional Learning (2016)

Smith, C., McGovern, G., Larson, R., Hillaker, B., & Peck, S. C.

In 2014, the Susan Crown Exchange (SCE) and the David P. Weikart Center for Youth Program Quality, set out to determine how out-of-school programs throughout the country can be more intentional about providing social and emotional skill development. The guide features 8 programs that have a proven track record of working with one of the hardest populations to reach: vulnerable and at-risk adolescents. The field guide sheds new light on how out-of-school programs can equip teens with valuable social and emotional skills.

 

Juvenile Justice

The Primary Factors that Characterize Effective Interventions with Juvenile Offenders: A Meta-Analytic Overview (2009)

Mark Lipsey

The meta-analysis presented in this paper was designed to test a broader range of intervention factors in a manner that allows for identification of both the general principles and the distinct intervention types associated with the greatest reduction in recidivism. These factors included a “therapeutic” intervention philosophy, serving high risk offenders, and the quality of implementation.

 

Improving the Effectiveness of Juvenile Justice Programs (2010)

Mark Lipsey, James Howell, Marion Kelly, Gabrielle Chapman and Darin Carver

The publication provides a brief history of criminal justice treatment philosophies and an overview of an evidence-based practices approach in the juvenile justice field. The report then summarizes Dr. Lipsey’s meta-analysis of juvenile justice programs, the SPEP tool, and policy considerations for state and local practitioners.

 

Juvenile Justice System Improvement: Implementing an Evidence-Based Decision-Making Platform (2017)

Mark Lipsey, Catherine Conley, Gabrielle Chapman, and Shay Bilchik.

The report summarizes some of the Department of Justice’s findings from the Juvenile Justice Systems Improvement Project (JJSIP) and the Juvenile Justice Reform and Reinvestment Initiative (JJRRI). This includes the implementation of an evidence-based decision-making platform consisting of validated risk and needs assessment tools, structured decision-making tools to match youth with the correct type of service, and evidence-based programs and services.

 

Child and Adolescent Mental Health

Understanding the Common Elements of Evidence-Based Practice: Misconceptions and Clinical Examples (2007)

Bruce Chorpita, Kimberly Becker, and Eric Daleiden

In this article, the authors summarize what a common elements approach to evidence-based practice would look like as well as potential barriers to dissemination, clinical applications of this approach, and common misunderstandings of this approach.

 

Identifying and Selecting the Common Elements of Evidence Based Interventions: A Distillation and Matching Model (2005)

Bruce Chorpita and Eric Daleiden.

The article proposes a new model where researchers can distill the intervention literature to derive profiles from evidence-based approaches. Practitioners can then match these profiles to individual clients based on client target problems as well as demographic and contextual factors. This model could facilitate improved understanding of similarities and differences among treatments, guide treatment selection and matching to clients, and point to new interventions based on the current research base.

 

Structuring the Collaboration of Science and Service in Pursuit of a Shared Vision (2014)

Bruce Chorpita and Eric Daleiden

The article identified methods that could preserve the strengths of the two major paradigms in children’s mental health, evidence-based treatments and individualized care models while expanding their applicability and impact. The article argues that future models for effective systems will involve increased attention to client and provider developmental pathways; explicit frameworks for coordinating people and the knowledge and other resources they use; and a balance of evidence-based planning and informed adaptation.

 

Mapping Evidence-Based Treatments for Children and Adolescents: Application of the Distillation and Matching Model to 615 Treatments from 322 Randomized Trials (2009)

Bruce Chorpita and Eric Daleiden

The authors used a distillation and matching model to examine 322 randomized clinical trials. They used coding to describe the practice elements included in each treatment protocol and empirically matched these elements to client factors. This study is the first to aggregate evidence-based treatment protocols empirically according to their constituent treatment procedures. The results point to gaps in the current child and adolescent treatment literature.

 

Other

 

Realist Randomized Controlled-Trials: A New Approach to Evaluating Complex Public Health Interventions (2012)

Bonell, C; Fletcher, A; Morton, M; Lorenc, T; Moore, L.

This paper discusses how randomized controlled-trials evaluate public health interventions and suggests methods for ensuring that these evaluations answer which interventions work for whom and under what circumstances.

 

A Component-Centered Meta-Analysis of Family-Based Prevention Programs for Adolescent Substance Use (2017)

Mark J. Van Ryzin, Cary J. Roseth, Gregory M. Fosco, You-kyung Lee, and I-Chien Chen

This paper provides a meta-analysis of family-based prevention programs for adolescents which focus on substance use. The analysis is centered around which common components are most effective in this policy area.