At the University of Colorado Boulder, we work with a team of implementation experts who have been helping schools and communities around the country implement evidence-based interventions (EBIs) for the past 20 years. When COVID-19 hit, our work was dramatically affected, as it was for all of you. Using previously collected process evaluation data, we were able to ask a critical question about how to move forward with one of the big changes in our new COVID world—training facilitators. Considering that many of the Healthy Teen Network followers and readers use evidence-based interventions (EBIs) in their work to promote healthy youth development, our question and study may be of interest to you:

Does it matter if EBI facilitators are trained in-person or online?

Research has shown that facilitators of EBIs implement best when they have pre-program training. Pre-program training is an efficient and effective way to orient facilitators to the background, theory, key practices, and values of an EBI, as well as provide opportunities to practice new methods.

LST is a school-based universal prevention program for middle school-age students and teaches that aims to reduce substance use, violence, and other behavioral health issues through teaching personal self-management skills (e.g., self-esteem, problem solving, coping with stress and anxiety), social skills (e.g., communication, building relationships, and assertiveness), and drug resistance skills (e.g., consequences of drug use, refusal skills). Like Botvin LifeSkills Training (LST) encourages in-person and on-site training that spans 1-2 days. This general recommendation for in-person training often presents economic and pragmatic challenges. In the COVID age, in-person training also presented health threats. Like everything from our work to social life, overnight, all our trainings moved from in-person to online.

Given how important we knew pre-program training was for implementation, we wanted to know if this shift in training modality would influence how well facilitators implemented the EBI. Thankfully, we had some data to shed light on this question.

What we did:

In a limited capacity, our project had begun offering online training options. We used the prior three years (2016-2019) of process evaluation data (collected by local classroom observers) to examine differences between teachers trained online versus in-person on four common measures of fidelity of implementation: 1) adherence to the curriculum (i.e., percent of curriculum points taught), 2) amount of time teaching the curriculum, 3) quality of delivery (e.g., teacher enthusiasm, knowledge, rapport with students), and 4) student responsiveness (student understanding and participation). These data included 989 LST teachers (107 trained online, 882 trained in-person) across 114 school districts, representing 296 schools in 14 states.

What we found:

• No differences were found on the percent of curriculum points covered, amount of time spent teaching, or student responsiveness.
• Online training was associated with lower quality of delivery.
  Ultimately, we identified three major takeaways:
• Online training builds competencies important for curriculum-based EBI implementation. Especially when in-person training is not possible, online training is a great option!
• Online training might not be a perfect substitute, though!
• Ensuring the inclusion of experiential learning activities (e.g., practice delivering content, receiving feedback on delivery) may be key to quality of delivery as online trainings for facilitators of curriculum-based EBIs evolve.

Some caveats and limitations should be stated regarding these findings.

First, online training served as a “backup” when in-person training was not possible. This resulted in the online training group being relatively small compared to the in-person training group, and reasons for some teachers’ participation in online trainings may have related to their motivation for implementing the EBI or characteristics of their environment (e.g., turnover). So, it’s possible that these reasons (not the online training itself) contributed to differences on quality of delivery.

Second, findings are not necessarily generalizable to all online trainings for EBIs. With the COVID-19 pandemic, EBIs quickly transitioned to online training and thus the format of online trainings is rapidly evolving; the online training examined in this study may not reflect the exact formats or versions that are currently in widespread use.

If you would like to read the study published in Prevention Science, you can find it here.