In the early months of the COVID-19 pandemic, child and youth mental health agencies shifted to delivering their services through virtual platforms. Our two organizations, the Ontario Centre of Excellence for Child and Youth Mental Health and Children’s Mental Health Ontario, worked together to evaluate the implementation of this shift to virtual care (Danseco et al, 2020). We define virtual care broadly, as any type of service delivered using telecommunication technology. This includes services and support provided over videoconference, phone, texting, and apps. We developed an evaluation framework informed by implementation science to obtain the perspectives of service users (clients including youth and families), service providers and agency leaders. We wanted to understand how clients and service providers experienced the transition to virtual care, the processes, facilitators and challenges in delivering virtual care. Exercises and resources are based on our evaluation which included the participation of 97 agencies and 192 youth and family members using online surveys, and interviews with 13 agency leaders and focus groups with 13 service providers. Service providers included social workers, psychologists, child and youth counselors from various types of mental health programs across Ontario. This presentation will provide participants with the steps to conducting a multi-level, mixed methods evaluation (including engaging stakeholders, developing the evaluation framework, conducting both qualitative and quantitative data analysis, developing recommendations for enhanced uptake of evaluation findings, and knowledge mobilization). Participants will learn how to use the implementation drivers (National Implementation Research Network, 2013), and the Consolidated Framework for Implementation Research (CFIR; Damschroeder et al., 2009) and the Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability (NAASS; Greenhalgh et al., 2017) to develop their questions and to analyze qualitative data. Resources will include a sample evaluation framework and checklist, sample interview questions and the sample CFIR codebook. We will also provide examples of evaluations from agencies, and a template for mobilizing evaluation findings. Our program evaluation toolkit will also be available as a downloadable resource, which includes various templates such as a matrix for developing process and outcome evaluations, budget templates, and stakeholder analysis.
- Describe the implementation frameworks that can be used for evaluating virtual services (e.g. NIRN implementation drivers, CFIR and NAASS)
- Use the Consolidated Framework for Implementation Research codebook for integrating perspectives from service users, providers and agency leaders
- Describe key elements in mobilizing evaluation findings with key stakeholders and enhance uptake of recommendations in improving virtual care