It is estimated that one in seven young people experience a mental illness and that about 70% of mental health challenges have their onset before the age of 18 [1, 2]. Anxiety disorders are the most common of these disorders affecting five percent of Canadian children aged 5–17 years old while 2.1% reported a mood disorder in 2019 [3]. Mental illness is one of the leading causes of disability among adolescents [1]. Despite the burden of mental health conditions in children and adolescents, data suggests that current mental health services are not meeting the needs of a majority of these children and youth. Only 44.2% of children experiencing a mental health condition received any services, revealing a large gap in services for children and youth mental health [4]. In Alberta, a Western Canadian province, the rates of emergency department visits for pediatric mental health concerns increased 35% between 2010 and 2020 [5], likely the result of a lack of availability of services for urgent mental health needs in the community [6]. The COVID-19 pandemic has further amplified the critical need for adequate mental health supports and services for children and youth [7, 8].
To address the needs of children and youth in Alberta, a new centre for children and adolescents with urgent mental health concerns, called “The Summit: Marian & Jim Sinneave Centre for Youth Resilience” (herein, The Summit) is opening in March 2023 in Calgary, Canada. The City of Calgary is located in the province of Alberta and is Canada’s third largest municipality, serving 1.3 million people [9]. The Summit will support children and youth (0–18 years old) with acute mental health concerns in a community setting. The centre will be operated by Alberta Health Services (AHS), the province-wide, publicly-funded health system in Alberta, Canada. The Summit has an integrated research program in partnership with the University of Calgary. This close partnership between academic researchers and the health system is supporting the development of The Summit as a learning health system (LHS).
As defined by Menear et al. (2019), an LHS is a “dynamic health ecosystem where scientific, social, technological, policy, legal and ethical dimensions are synergistically aligned to enable cycles of continuous learning and improvement to be routinised and embedded across the system, thus enhancing value through an optimised balance of impacts on patient and provider experience, population health and health system costs.” AHS is facing rising health care costs and is therefore moving towards a model of Value-Based Health Care (VBHC), where the purpoted aim is to offer patients the best possible outcomes weighed against the cost of providing services [10, 11]. The implementation of a LHS is seen as a pathway towards operationalizing VBHC and thus, establishing a LHS has become a priority for AHS [12].
Foundational to a LHS is the rapid executing of learning cycles, where data are generated within clinical practice and transformed to knowledge and that knowledge is rapidly applied back into clinical practice [11]. A key source of data in a learning health system is patient-reported outcomes (PROs), gathered with patient-reported outcome measures (PROMs). PROMs are validated questionnaires filled out by the patient about their health status, quality of life, or well-being [13]. PROMs provide valuable data about outcomes from the patient’s perspective that can be used within an LHS to answer questions regarding treatment effectiveness, disease trajectories, and the value of care provided [14]. PROMs also provide valuable information for the clinician and patient’s own learning, for example, whether a particular treatment is having the intended effect on a patient’s symptoms. The use of PROMs in routine clinical care can improve communication between patient and clinicians, enhance share-decision making, self-management and patient outcomes [15, 16].
Despite the value of PROMs within a health system, their successful implementation into routine clinical care is complex [17, 18]. It requires a significant investment of time and resources, as well as the development of a supportive information technology infrastructure [17, 19]. Commonly cited challenges to their implementation include finding appropriate PROMs (i.e., relevant to patients’ needs, feasible to use, and adding value to the clinical encounter), designing PROMs data collection processes around clinical workflows, getting buy-in from clinical staff, and creating a culture that is supportive of learning and quality improvement [17]. In child and youth mental health care, there are additional challenges, including finding PROMs that have been validated for their use in children and youth, developing data collection platforms and workflows that allow the collection of responses from both parents and children [20, 21]. There are also additional ethical questions to navigate, such as whether the youth, one caregiver, both parents or all should be responding to PROMs, whose responses will be more strongly weighted, whether a parent should respond to a PROM about their child without their child’s consent, and at what age a child has the right to keep their PROMs data private from their caregivers [21,22,23].
PROMs will be embedded into routine clinical care processes at The Summit when it opens. At the initial stages, the focus on PROMs will be their use as an intervention to enhance learning for the patient and clinician about the patients’ function and responses to treatment. PROMs data used for learning at the meso and macro levels will begin once sufficient data are collected for planned research projects and quality improvement, although the success of these activities will depend on the quality of PROMs data that is collected. Given the complex interactions between technical, organizational, personal and familial factors that may impact how effectively PROMs are collected and used for individual and family clinical care, it is crucial to assess and address unidentified or unresolved barriers to effective implementation in order to be successful in the implementation of PROMs at The Summit. The assessment will inform refinements to the components of the PROMs intervention and the implementation strategies, optimizing PROMs data collection and clinical use at the centre.
Purpose
This paper describes a protocol for the formative evaluation of the implementation of PROMs for clinical care within The Summit. The purpose is to optimize the collection and use of PROMs. The evaluation objectives are to:
-
1.
Assess the implementation progress to date (penetration, fidelity, appropriateness, and adoption)
-
2.
Identify barriers and facilitators to implementation and sustainability of PROMs at The Summit
-
3.
Explore patient, caregivers and clinician experiences of PROMs in clinical care
Context
The Summit is a stand-alone facility which aims to serve children and youth ages 0–18 years old with emerging or worsening mental health concerns. It will have three main services: (1) a walk-in service for urgent mental health concerns; (2) a day program to support youth transitioning from a hospital admission back to their home; and (3) intensive community treatment services, designed to prevent the need for hospitalization, which provides youth with escalating mental health concerns with short-term, intensive therapy, consisting of group, family and individual therapy, and psychiatry services.
In partnership with researchers at the University of Calgary, The Summit has an embedded, patient-oriented research program with three functions: (1) generating new knowledge from data collected from patients at The Summit; (2) testing new interventions for children and youth with depression and anxiety; and (3) implementing and evaluating effectiveness of interventions that are not widely available to children and youth in Calgary. Additionally, the socioeconomic (i.e., cost-effectiveness and policy) impact of the interventions offered will be assessed.
Intervention description
Measurement-Based Care (MBC) is a commonly used term to describe the routine clinical use of PROMs in mental health services [22]. It is accepted that MBC consists of three main steps: (1) administering a symptom, outcome, or process measure (i.e., PROM) ideally before a clinical encounter; (2) clinician and patient review of PROM data; (3) collaborative re-evaluation of the treatment plan informed by data [24].
MBC will be a standard part of care at The Summit. Three PROMs will be administered across all services at the Centre, the acute version of the Pediatric Quality of Life Inventory Generic Core Scales (PedsQL), the Revised Children’s Anxiety and Depression Scale (RCADS-25), and Columbia-Suicide Severity Rating Scale (C-SSRS). These measures were chosen based on the International Consortium for Health Outcomes Measurement’s Set of Patient-Centered Outcome Measures for Children and Young People with Anxiety and Depression, including OCD and PTSD, informed by the PROMs available within the provincial electronic health record (EHR), and in collaboration with patients, caregivers, and researchers [25].
The acute version of the PedsQL is a 23-item questionnaire with a 7-day recall period. It has 4 scales (physical, emotional, social, school functioning) and three summary scores (Physical Health Score, Psychosocial Health Score and Total Scale Score) [26]. It has self-report forms for children in 3 different age groups and parent-report forms for 4 different age groups [26]. RCADS-25 is a 25-item questionnaire for children and youth ages 8–18 with 2 subscales (anxiety and depression) and a total scale score which can be converted to norm-based T scores [27]. It has self-report and parent-report forms. The C-SSRS is a 7-item self-report questionnaire for youth ages 10 and older, which classifies individuals as high, moderate or low risk for suicide [28]. The RCADS-25 and C-SSRS have demonstrated good reliability and evidence of their validity in pediatric mental health populations [27, 28]. The reliability and validity of the PedsQL in this population has not been reported, however in the general population it performs well and is widely used within Alberta’s pediatric health system [29].
Figure 1 illustrates PROM data collection timepoints for each service. For walk-in services, PedsQL will be administered routinely at intake, with the intake process flagging whether RCADS and C-SSRS need to be administered. For the 2-week day hospital program, all three PROMs will be administered at intake and at discharge. For intensive community treatment services, the three PROMs will be administered at intake, midpoint of treatment (after 2 weeks) and at discharge. In each case, the PROMs will be automatically scored and represented graphically (along with prior PROM scores) in a downloadable summary document which also contains patients’ responses to each item. The summary document will be imported into the patients’ EHR by the intake coordinator for the service. During the clinical encounter, the clinician will access patient scores in the EHR and review them with the patient. Together they will discuss patient concerns highlighted by the PROM and re-evaluate the treatment plan. Whenever possible, PROMs will be administered to both patients and caregivers. A clinician-reported measure, the Children’s Global Assessment Scale (CGAS), will be also be collected at each PROM collection timepoint in the day program and intensive community treatment services, as it is part of the standard of care for children and youth mental health services in Alberta [30].
Patient-reported outcome measures in the three services at The Summit
Implementation process
The implementation of PROMs at The Summit was guided by the Quality Implementation Framework (QIF) [31]. The QIF is an implementation process model which was designed to guide the “how-to” processes of implemenation [31]. It has 14 steps divided into four phases: (1) Initial considerations regarding the host setting, (2) Creating a structure for implementation, (3) Ongoing structure once implementation begins, (4) Improving future applications [31].
We formed a MBC working group consisting of operational leadership and academic partners (clinical and health systems researchers) 18 months prior to the opening of The Summit. The working group assessed the context; adapted the MBC intervention to The Summit; developed the information technology structure to collect, use and record PROMs in the patient EHR; and designed the implementation approach. The implementation strategies were developed based on existing PROMs implementation guidelines [24, 25, 32, 33]. We engaged youth and caregivers through The Summit’s advisory councils in designing the PROMs processes, selecting the PROMs to use and informing the implementation strategies.
Once an opening date for The Summit was finalized, an implementation team was created, consisting of two clinical leads from The Summit and 4 researchers. The implementation team finalized the PROMs clinical workflows and created a detailed plan for implementation. Once The Summit opens and PROMs collection begins, onsite technical and clinical support will be offered through the implementation team.
This formative evaluation is part of phase three of the QIF. We used the Implementation Research Logic Model (IRLM) to specify the core elements of the implementation design [34]. The IRLM for this project details the determinants, strategies, proposed mechanisms of change and outcomes for the MBC implementation project and is available as supplementary online material (Supplementary File 1).
Add Comment