Background: Primary care clinics present challenges to implementing evidence-based psychotherapies (EBPs) for depression and anxiety, and frontline providers infrequently adopt these treatments. The current study explored providers’ perspectives on fidelity to a manualized brief cognitive behavioral therapy (CBT) as delivered in primary care clinics as part of a pragmatic randomized trial. Data from the primary study demonstrated the clinical effectiveness of the treatment and indicated that providers delivered brief CBT with high fidelity, as evaluated by experts using a standardized rating form. Data presented here explore challenges providers faced during implementation and how they adapted nonessential intervention components to make the protocol “fit” into their clinical practice. Methods: A multiprofessional group of providers (n = 18) completed a one-time semi-structured interview documenting their experiences using brief CBT in the primary care setting. Data were analyzed via directed content analysis, followed by inductive sorting of interview excerpts to identify key themes agreed upon by consensus. The Dynamic Adaptation Process model provided an overarching framework to allow better understanding and contextualization of emergent themes. Results: Providers described a variety of adaptations to the brief CBT to better enable its implementation. Adaptations were driven by provider skills and abilities (i.e., using flexible content and delivery options to promote treatment engagement), patient-emergent issues (i.e., addressing patients’ broader life and clinical concerns), and system-level resources (i.e., maximizing the time available to provide treatment). Conclusions: The therapeutic relationship, individual patient factors, and system-level factors were critical drivers guiding how providers adapted EBP delivery to improve the “fit” into their clinical practice. Adaptations were generally informed by tensions between the EBP protocol and patient and system needs and were largely not addressed in the EBP protocol itself. Adaptations were generally viewed as acceptable by study fidelity experts and helped to more clearly define delivery procedures to improve future implementation efforts. It is recommended that future EBP implementation efforts examine the concept of fidelity on a continuum rather than dichotomized as adherent/not adherent with focused efforts to understand the context of EBP delivery. (Continued on next page) * Correspondence: Joseph.Mignogna@va.gov VISN 17 Center of Excellence for Research on Returning War Veterans, Central Texas VA Health Care System, 4800 Memorial Drive (151C), Waco, TX 76711, USA Department of Psychiatry and Behavioral Sciences, Texas A&M University Health Science Center, Temple, TX, USA Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Mignogna et al. Implementation Science (2018) 13:76 Page 2 of 11 (Continued from previous page) Trial registration: ClinicalTrials.gov, NCT01149772 Keywords: Cognitive behavioral therapy, Fidelity, Adaptation, Implementation, Integrated primary care, Depression, Anxiety, Qualitative methods, Pragmatic trial Background adaptation tension is a notable challenge for all settings but Implementing evidence-based psychotherapies (EBPs) is is particularly important for primary care settings where a priority for the largest healthcare organization in the few EBPs currently exist. USA, the Veterans Health Administration (VHA). Over This article explores intervention adaptation data from the past decade, VHA has heavily invested in imple- a pragmatic clinical trial of an EBP for integrated pri- menting and disseminating EBPs for many conditions, mary care settings. Using the Dynamic Adaptation including posttraumatic stress disorder, depression, in- Process (DAP) model, a series of qualitative interviews somnia, and chronic pain [1–7]. However, efforts have with mental health providers were conducted to explore largely targeted specialty mental health rather than men- their efforts to deliver a brief CBT intervention while tal health integrated primary care settings, and chal- balancing the tension between adhering to the treatment lenges remain regarding the broader use of EBPs and protocol and meeting the needs of their patients and whether EBPs are delivered with fidelity . clinical settings. Notably, the DAP model is consistent The primary care arena is different in numerous ways with the “flexible fidelity” view of intervention adapta- from traditional mental health care settings (e.g., tion , highlighting the need to better understand faster-paced environment, briefer treatment sessions, “how to facilitate delivery of EBPs with appropriate ad- briefer courses of treatment) and presents unique chal- herence and competence, while allowing for adaptations lenges to the delivery of EBPs . Brief EBPs have been that do not interfere with core elements” (, p. 2). developed to fit the primary care setting, and preliminary The DAP model posits that adaptation occurs across evidence suggests that brief cognitive behavioral therapy four phases (Exploration, Preparation, Implementation, (CBT) and problem-solving therapy for depression and and Sustainment). During Implementation, five sources anxiety can be effective in primary care clinics [10–12]. of ad hoc adaptations can occur to peripheral compo- Despite the availability of these EBPs that were specifically nents of the treatment or its delivery, namely, developed to address known implementation barriers, evi- patient-emergent issues, provider skills and abilities, dence suggests psychotherapy providers in primary care available resources, provider knowledge, and commonly deliver only isolated components of EBPs . organizational changes. Compared to other models Focused implementation efforts are still needed to im- examining the fidelity-flexibility tension (e.g., modified prove provider use of EBPs for depression and anxiety. fidelity framework ), the DAP model provides a Providers often struggle to incorporate EBPs that are meaningful and face-valid model for capturing the ten- protocol-based and largely developed in non-clinical set- sions our study clinicians reported experiencing in their tings for research purposes. They often report difficulty efforts to balance fidelity with adaptation while imple- “translating” EBP protocols to fit their practice while retain- menting brief CBT in primary care. ing the empirically supported nature of the treatment itself. Brief CBT is designed to better fit the unique needs of Delivering a treatment as it was intended to be delivered the primary care setting. However, implementation chal- versus the need to change the intervention or how it is de- lenges remain. Data from this study provide information livered to improve its “fit” is commonly referred to as fidel- about the specific ad hoc adaptations clinicians used to im- ity versus adaptation (i.e., flexibility). Fidelity refers to how plement brief CBT in a mental health integrated primary closelyaproviderskillfullydelivers the treatment compo- care setting. Knowledge of the scope of potential adapta- nents believed essential to attaining intended treatment ef- tions will inform implementation planning for brief EBPs to fects . These essential components of treatment are better balance intervention fidelity with real-world imple- commonly referred to as “core” . Some advocate for a mentation delivery. Additionally, the current study tests the “flexible fidelity,” whereby successful implementation re- utility of the DAP model for understanding psychotherapy quires providers to deliver an EBP with high fidelity; how- treatment adaptation during implementation ever, intentional adaptations to “peripheral” components (i.e., any treatment components not viewed as essential to Methods obtaining intended treatment effects) are done to maximize Brief cognitive behavioral therapy intervention the “fit” of an intervention in the context it is delivered and The brief CBT intervention was part of a hybrid type 2 thereby promote sustainability [15, 16]. Thefidelityversus study that evaluated the effectiveness and Mignogna et al. Implementation Science (2018) 13:76 Page 3 of 11 implementation of brief CBT in the primary care setting current study was substantially higher than in typical at two VA medical centers (here referred to as site 1 and care. Patients could choose to receive treatment by site 2 ). The intervention targeted primary care pa- phone or in-person. Providers typically delivered the first tients with heart failure and/or chronic obstructive pul- session in person, unless the patient specifically re- monary disease, with clinically elevated symptoms of quested a phone session; however, subsequent sessions anxiety and/or depression [19–21]. The study used the (i.e., sessions 2–6) were deliverable through either mo- Reach, Effectiveness, Adoption, Implementation, and dality. More than half of sessions 2–6 were by phone Maintenance (RE-AIM) framework to evaluate key ef- (60.3%) . fectiveness (e.g., lower rates of depression and anxiety) Main study outcomes and a complete description of and implementation outcomes (e.g., fidelity to treatment the parent study can be found elsewhere [18, 21]. Add- delivery). Essential, or “core,” components of our brief itionally, the clinician manual and patient workbook are CBT include (1) delivery of the intervention in four to freely available online . All study procedures were six sessions, approximately 30–45 min (consistent with approved by each site’s respective Institutional Review the session duration and number of sessions of other Board and VHA Research & Development Committee. EBPs designed for delivery in this setting [10–15]) over a 4-month period and (2) delivery of brief CBT sessions Implementation strategy with acceptable adherence and competency scores, as A multicomponent implementation strategy was guided by determined through the Adherence and Competence the evidence, context, and facilitation domains of the Pro- Evaluation (ACE) rating forms described in detail below moting Action on Research Implementation in Health Ser- and attached as an appendix. A minimum of four ses- vices (PARIHS) framework to support adoption of brief sions within 4 months was defined a priori as the criter- CBT. To build providers’ knowledge of brief CBT and its ion for treatment completion. Four sessions allowed effectiveness (i.e., evidence domain), the implementation providers and patients flexibility to tailor treatment strategy included a modular-based online providers’ train- while seeking to deliver care in an efficient manner. Ex- ing program and ongoing audit and feedback from a CBT tending therapy beyond six sessions or 4 months was expert on audio-recorded treatment sessions to promote deemed an acceptable treatment adaptation. Import- provider adherence and competence in delivering key mod- antly, in other research using the parent study data, a ule components. Program directors and clinical champions dose-response effect from a number of treatment ses- served as internal facilitators supporting and promoting the sions was not found . use of brief CBT in the primary care context. Study team Sessions 1 and 2 were required modules that explored memberswho were outsidethe primarycaresetting pro- the connection between physical and mental health vided external facilitation to discuss providers’ challenges problems and improving health through action planning implementing brief CBT and problem solve logistical issues and goal setting. For sessions 3 through 5, providers through individual and group mentoring meetings with worked with patients to select three of four skill-focused providers [18, 25]. External facilitation allowed the project modules focusing on exercise and nutrition, changing team to address practice challenges across sites, while in- negative thought patterns, and behavioral activation or ternal facilitation provided a detailed within-site approach. relaxation. The final treatment session (i.e., session 6) Participation in the group mentoring meetings or using ses- reviewed and consolidated treatment progress. Patients sion note templates for patients were optional implementa- were asked to practice skills between sessions through tion supports available to providers. homework exercises and were offered two monthly Audit and feedback was implemented using ACE rating “booster calls” following the final session to review and forms, a standardized evaluation of session audio record- support maintenance of skills acquired during treatment. ings (see Additional file 1). ACE rating forms provided a Of 180 patients randomized to brief CBT as part of checklist to ensure providers’ adequate delivery of core this pragmatic trial, 63.3% (n = 114) completed four or components of each brief CBT session. The CBT expert more sessions, 51.7% (n = 93) completed five or more used these rating forms to inform decisions about overall sessions, and 34.4% (n = 62) completed six or more ses- adherence and skill ratings for each reviewed session . sions. This represents a higher receipt and number of Scores were rated from 1 to 8 for both the Adherence (i.e., psychotherapy sessions than are typical for the inte- delivery of essential components of each treatment session grated primary care setting, with prior literature report- identified on the ACE rating form) and Competence (i.e., ing 61% only received one session . Further, given a ability of the provider to skillfully deliver the core compo- prior report that only 27% of patients newly diagnosed nents of the brief CBT manual in a manner that promotes with depression, anxiety, or posttraumatic stress disorder rapport, efficient use of session time, and relevance of the in the VHA received at least one session of psychother- treatment to the patient’s clinical needs) subscales, with apy , patients’ receipt of psychotherapy in the 4–5 scores anchored as “moderately” adherent/competent Mignogna et al. Implementation Science (2018) 13:76 Page 4 of 11 and 6, 7, and 8 classified as “good,”“very good,” and “ex- including licensed psychologists, psychology fellows and in- cellent,” respectively . terns, licensed clinical social workers, and physician assis- Consistent with the DAP model , providers re- tants (see Table 1). Providers were directly recruited for the ceived ongoing feedback from a CBT expert, using this parent study during staff meetings or through contacts in audit and feedback process, approximately once every primary care clinics. Providers gave written consent to par- 6 months during implementation. In addition to moni- ticipate in the parent study and subsequently verbally toring and providing ongoing feedback to clinicians to assented to the summative evaluation qualitative interview. promote fidelity to the core treatment components each Providers were contacted by email to schedule interviews at session, the CBT expert also provided guidance about a convenient time. One provider did not respond to our re- the acceptability of any treatment adaptations. While quest for an interview. Eighteen of the 19 participating pro- scores of 6 or 7 were deemed to be indicative of “high fi- viders (95%) completed the interview between November delity,” scores of 4 or less were deemed the minimally 2012 and April 2014 to document their experiences imple- acceptable ACE rating. On the rare occasion a provider’s menting the brief CBT intervention (n =11 at site 1 and n ratings fell below a score of 4, additional audit and feed- = 7 at site 2). Interviews were conducted over the phone (n back was provided by the CBT expert until acceptable = 16) or in person (n =2) and were also digitally audio re- scale ratings were achieved. The CBT expert permitted corded and transcribed. Interviews lasted between 30 and and sometimes encouraged providers to adapt peripheral 50 min (M =41 min). components of the modules to improve the fit of brief Development of the semi-structured, open-ended inter- CBT into their clinical practice while concurrently main- view guides was informed by the PARIHS and RE-AIM taining high fidelity to key components. To determine frameworks [27–30]. In addition to our use of PARIHS in whether a treatment adaptation was acceptable, the CBT guiding implementation in the parent study, our rationale expert consulted with the treatment developers. For ex- for selecting the PARIHS framework to guide the ample, using two sessions to deliver a session module was viewed as acceptable. In contrast, adaptations that Table 1 Provider characteristics interfered with delivery of core treatment components Gender Female 15 (83.3%) (as detailed for each session in the ACE rating scales; see Male 3 (16.7%) supplemental material) were viewed as unacceptable. For Type of provider Licensed 5 (27.8%) example, the CBT expert discouraged providers from an- psychologists cillary discussions and being “off task” (e.g., storytelling Psychology 6 (33.3%) by patients) as this detracted from the providers’ time to fellows accomplish essential components of the treatment. Psychology 3 (16.7%) Providers received expert feedback on an average of seven interns audio recorded sessions. Of 602 audio-recorded treatment Licensed clinical 2 (11.1%) sessions, 23% (n = 137) were audited . Afterward, pro- social workers viders received feedback by phone and/or email. Feedback Physician 2 (11.1%) was provided in the spirit of professional development and assistants growth rather than an identification of “problematic” or “in- Time as a mental health provider (not < 1 year 11 (61.1%) appropriate” behaviors. For example, experts attempted to including training) 1–3 years 2 (11.1%) identify positive as well as “developmental” behaviors to en- 4–5 years 3 (16.7%) courage growth and development . Providers demon- > 10 years 2 (11.1%) strated high fidelity ratings, with average adherence and Time affiliated with primary care mental < 1 year 12 (66.7%) competence of 6.7 and 6.2, respectively . They repre- health integration sented a fairly homogenous group, with limited variability 1–2 years 6 (33.3%) among provider fidelity ratings and no providers scoring Length of time conducting cognitive < 1 years 13 (72.2%) less than an average of 6. Further, provider differences (e.g., behavioral therapy in mental health 1–3 years 1 (5.6%) practice (not including training) professional discipline) did not statistically impact treat- 4–5 years 3 (16.7%) ment outcomes . > 10 years 1 (5.6%) Length of time working with chronic < 1 year 9 (50.0%) Qualitative summative evaluation and/or complex medically ill patients Data collection 1–3 years 4 (22.2%) The Consolidated Criteria for Reporting Qualitative Studies 4–5 years 3 (16.7%) guides our reporting of qualitative data collection, analysis, 6–10 years 1 (5.6%) and findings . Providers interviewed were a diverse > 10 years 1 (5.6%) group of VHA primary care mental health professionals, Mignogna et al. Implementation Science (2018) 13:76 Page 5 of 11 development of the interview guide was influenced by the In phase two, data deductively coded as “implementa- parsimonious account of factors it offers as important for tion” were extracted from the Atlas.ti database by one implementation success. Similarly, RE-AIM offers a parsi- analyst (LM) to further explore providers’ perspectives monious account of implementation outcomes of interest. on brief CBT implementation. The analyst (LM) re-read This simplicity and strong empirical basis served as the these interview segments and pile sorted them by hand primary reasons we selected the PARIHS framework and into categories, i.e., subcodes comprising similar topics RE-AIM model over alternatives  to guide the devel-  that emerged inductively from the data. Data coded opment of our focused yet comprehensive interview guide. as “implementation” broadly encompassed aspects of Open-ended questions covered the following: (1) pro- treatment fidelity, for example, level of provider adher- viders’ general experiences with the brief CBT interven- ence to the treatment protocol, how providers modified tion, (2) beliefs about evidence-based psychotherapy and and/or deviated from the protocol, and how providers manualized treatments, (3) fit of brief CBT in the primary chose to deliver the treatment (i.e., mode of treatment care setting, and (4) perceived outcomes, implementation, delivery). After further review of the codebook and feed- and lasting potential of brief CBT (see Additional file 2). back from the analysis team, it was decided that one of A clinical psychologist experienced in qualitative methods our inductive codes from phase one represented an as- (JH), but not directly affiliated with the study, conducted pect of implementation, and it, therefore, became a sub- interviews to maintain objectivity in data collection. Pro- code in this phase of our analysis. To optimize validity, viders were interviewed after they completed participation the second analyst (JM) audited the findings to offer as study therapists. On average, providers interviewed feedback and suggestions for refinement and to provide were assigned between 5 and 25 patients (M = 9.9 pa- consensus on identified subcodes. A detailed review of tients) before completing the interview, so their perspec- these subcodes illustrated various ways providers negoti- tives on implementing brief CBT are based on a broad ate the tension between fidelity to and a need to display range of clinical experiences. flexibility in delivering brief CBT in the primary care set- ting and informed our description of the themes below. Analysis Results Qualitative analysis occurred in two phases: phase one The data collected through the individual qualitative inter- was largely deductive, and phase two was inductive. In view data is for the purpose of better understanding pro- phase one, a medical anthropologist (LM) and a clinical viders’ experiences responding to tensions arising from psychologist who served as the study external facilitator adaptation-fidelity concerns. The following themes detail (JM) employed directed content analysis . The di- providers’ experiences navigating the fidelity versus flexi- rected content analysis utilizes predefined, a priori, cat- bility tension during implementation, including descrip- egories to guide analysis. In this phase of analysis, tions of adaptations made to peripheral components of evidence, context, and facilitation (the main domains of the brief CBT or how it was delivered. The DAP model’s the PARIHS framework), and the Reach, Effectiveness, provider skills and abilities, patient-emergent issues, and Adoption, Implementation, and Maintenance domains available resources ad hoc adaptations, which occur dur- (from the RE-AIM framework) served as our predefined ing the implementation phase, provide an overarching deductive codes. In addition to these predefined coding framework to better understand and contextualize these categories, analysts also identified additional categories themes . through a posteriori (inductive) coding. Analysts con- structed a codebook containing code names and brief Adaptations associated with provider skills and abilities descriptions with key elements pertaining to each code. We conceptualized “provider skills and abilities” as a The principal investigator of the parent study (JC) and provider’s capacity to adjust his or her approach to deliv- the interviewer (JH) provided feedback on the codebook, ering brief CBT based on the level or amount of patient leading to further refinement. Twelve transcripts were engagement with the treatment protocol (i.e., how the coded by both analysts, who regularly met in consensus provider drew upon his/her clinical skills to respond to meetings to compare results, discuss and resolve dis- the patient’s desire to engage in brief CBT). For example, crepancies, and revise the codebook. Once the analysts providers describe how some patients required more reached consistency in assigning codes and reached con- guidance from providers than others when choosing sensus on the codebook, they divided the remaining six which of the four skill-building modules (to be used dur- transcripts and coded independently. Atlas.ti (v. 6.2, ing sessions 3–5) aligned with their treatment goals. Atlas.ti Scientific Software Development GmbH, Berlin, One provider took a more hands-on approach by having Germany) qualitative software facilitated data manage- his/her brief CBT patients read about the modules and ment and coding. then choose what appealed to them. This provider also Mignogna et al. Implementation Science (2018) 13:76 Page 6 of 11 went on to say that if the patient “didn’t really respond” among this veteran population), and deaths of friends or after reading about the modules, she/he adjusted the ap- family. Because of these issues, patients would often go proach by talking more with the patient about his/her off topic during brief CBT sessions. Providers reframed needs, and then “pushed a little bit more to say, ‘You or redirected patients back on topic or found ways to ac- know…where…do we stand now?’” (K108). Sensing that knowledge and address these concerns, in part, if they the patient desired to be more self-directed when select- were brought up in treatment sessions. Providers also in- ing modules, another provider described adjusting the corporated patients’ broader life issues into their brief approach to a more hands-off style. However, this pro- CBT treatment goals or helped patients connect these vider noted that although this patient preferred to “take other life issues to skills they were learning in the brief charge” of module selection, it was atypical for patients CBT modules. Alternatively, providers offered to provide to engage with the protocol in this manner (i.e., most resources for additional help outside the context of the providers had to take the more guided hands-on ap- brief CBT intervention (e.g., informing the patient about proach) (C12). additional forms of mental health treatment, referring One area of brief CBT treatment that posed a chal- the patient back to the primary care provider to address lenge to “provider skills and abilities” was patient en- these concerns), and the option to talk about the issue at gagement in the between-session homework exercises. the end of the session so as not to dismiss the patient’s Patients were asked to complete homework exercises to real-life concerns. One provider attributed the emer- reinforce their skills learned during the sessions, but gence of these off-topic issues to a simple, they did not like this aspect of the brief CBT protocol. non-standardized, quality of life assessment patients Patient dislike of the homework exercises challenged completed with their providers during the first session. providers’ abilities to deliver effective brief CBT. One This assessment asked patients if issues such as physical provider noted how they set the expectation at the be- health, spirituality, finances, relationships, and emotional ginning of treatment that homework completion was im- health are sources of stress, which the provider felt may portant for treatment success. However, when patients have signaled to patients that these issues are appropri- still did not complete homework, the provider needed to ate topics to talk about during brief CBT. become more “emphatic” with them about the home- However, not all issues could be redirected or tied work assignments as treatment continued (C03). back to patients’ treatment goals. One provider de- Responding to the non-completion issue, another pro- scribed a pressing issue that required a complete devi- vider felt a “motivating rationale” was needed to help ation from the brief CBT protocol: “I had a patient boost patient engagement with homework (C02). whose wife died, and so …I had to step back and do “Provider skills and abilities” also encompassed a pro- more a … different type of therapy on that particular vider’s aptitude to draw on his/her own clinical experi- day” (C09). Providers also reported commonly feeling ences and adapt module content when needed. For the need to address patient mental health concerns be- example, providers demonstrated flexibility when deliv- yond the intended use of the treatment. For example, ering module content focused on teaching patients how one provider felt “constricted” by the manual and de- to do deep breathing exercises. One provider described scribed how a patient needed more than what the brief his/her preference for focused breathing exercises versus CBT intervention could provide. The patient had post- the deep breathing approach outlined in the brief CBT traumatic stress disorder that caused anxiety and phys- protocol. The provider preferred the former approach, ical symptoms, and the patient’s hypervigilance noting evidence that deep breathing exercises could trig- interfered with his/her ability to go for walks alone. The ger a distress response in pulmonary patients. Another provider “wanted to spend a little bit more time on that provider mentioned adapting how breathing was taught aspect of” the patient’s condition and did so by slightly by using their “own version,” that is, explaining the skill expanding on the behavioral activation session to help differently than what was in the providers’ manual the patient “problem solve around that barrier” (C10). (K102). Simplifying language used in the treatment man- ual, and learning how to better pace content delivered Adaptations associated with available resources during treatment sessions, also demonstrated providers’ Providers made several adaptations to maximize time, abilities to flexibly adjust to the manualized protocol. which was a valuable resource in the primary care clinics. Brief CBT was designed to be delivered within Adaptations associated with patient-emergent issues 35–40 min; in contrast, standard CBT is typically deliv- Providers noted how brief CBT sessions did not expli- ered in 45–50 min sessions. Thus, providers pared down citly offer skill-building modules to address patients’ session content by focusing on only one teachable skill, broader life and clinical concerns such as employment, summarized portions of module content more thor- relationships, finances, traumatic experiences (common oughly discussed in the patient treatment manual to Mignogna et al. Implementation Science (2018) 13:76 Page 7 of 11 ensure that content was understood, and delivered a noted: “[I]t’s hard to get them back in [for the missed skills module over two sessions instead of one to ad- treatment session], you know. It would probably take an- equately cover session content. At the outset of the other six weeks before I would have an opening like that treatment, the treatment developers determined that, at available again” (C04). To compensate for missed ses- the clinician’s discretion, a module could be delivered sions, providers from site 1 described adding an add- over two sessions instead of one. Consequently, if clini- itional session to the appointment block, booking the cians did this, it may have reduced the content reviewed patient into a slot normally held for primary care during treatment; however, it was viewed as important walk-in patients, or scheduling the patient into another to the clinician to maintain focus on a particular module slot normally held for something else (e.g., grand over two sessions. rounds). One provider felt it necessary to extend treatment by a Delivering brief CBT to patients by phone was a re- few minutes to facilitate the development of the thera- source providers could utilize during implementation. peutic relationship for medically and psychologically However, provider viewpoints differed on the impact of complicated patients. This provider believed that, al- phone-based CBT on treatment fidelity. While some though it was possible to deliver session content in the providers felt phone-based CBT increased access to care, designated timeframe, an “extra ten minutes would just resulted in fewer distractions during treatment, and was allow me to [say] okay. You know, ‘What are your dogs’ efficient (i.e., cut down on “chit chat” that can occur names?’…. ‘Oh, so your action plan this week is to go during in-person visits), others believed adapting brief out and take Bob and Sue for a walk.’ You know, and so CBT to phone-based delivery could hinder treatment fi- it just brought it [the treatment session] home for them” delity. Providers believed seeing the patient in person (K03). Providers also commonly remarked about talk- allowed them to develop a better therapeutic relation- ative patients that often wanted to share stories during ship and better assess his/her well-being via nonverbal treatment sessions that shortened the time available to forms of communication. Phone-based delivery gave complete session key components. Providers spoke providers less flexibility to provide patients with add- about slight modifications they made during the session itional copies of worksheets in the treatment workbooks. to address this scenario in treatment that included ask- One patient was asked to move to a more private loca- ing patients to share these stories at session end, if time tion due to too many distractions in the background, ac- remained. Of note, while the brief CBT was designed for cording to one provider. Some treatment skills were also delivery in 30 to 45 min, the fidelity rating scales did not reported to be more difficult to teach over the phone explicitly indicate provider strict adherence to this ses- (e.g., imagery relaxation). One provider articulated how sion time limit. she/he felt patients did not perceive phone sessions as Providers described the difficulty for both providers an “official” appointment (some patients were more in- and patients of being unable to deliver the intervention clined to miss phone than in-person sessions). Also, pro- on a weekly basis (a core component of treatment) when viders questioned the efficiency of phone-based brief patients no-showed due to life circumstances such as CBT because the delivery of session content went faster caregiving responsibilities, work, and medical comorbidi- by phone (suggesting less comprehension and integra- ties. One provider noted: “I think having more time in tion into a patient’s life) and the belief that more could between sessions makes it harder for them (i.e., patients) be accomplished face-to-face where providers could see to track what’s going on in treatment,” and consequently written homework exercises the patient completed. both providers and patients have “... trouble remember- ing what we talked about last session” (C06). One pro- Discussion vider described backtracking to review material from the This study reports findings from individual qualitative previous session, and how “it didn’t feel right because in interview data collected from 18 providers describing that first session you’re obviously establishing some level their experiences with striking a balance between fidelity of rapport. …[I]t’s not the same as when I just have and flexibility when implementing an EBP (brief CBT) in somebody (a regular primary care patient) walk into my primary care for medically ill veterans with clinically ele- office. …Because I have to stick to the manual, and vated depression and/or anxiety symptoms. Although there’s certain things I have to say, and certain things I providers maintained high fidelity to the brief CBT have to get through” (K102). protocol, they reported challenges and ad hoc adapta- Block or advanced scheduling was an available re- tions to improve the fit of the treatment for their clinical source at site 1 that promoted weekly session delivery, practice. Interview data supported empirical data that given how busy providers’ schedules were , and was providers were faithful to the brief CBT protocol but re- part of the implementation strategy. Despite this strat- ported strong tensions between the need for flexibility egy, if a patient missed an appointment, one provider while delivering the treatment to ensure effective use in Mignogna et al. Implementation Science (2018) 13:76 Page 8 of 11 real-world care settings . The delivery of EBPs is to make it “work.” Providers described the challenge of complex and involves the need to deliver a treatment ac- teaching patients tangible skills to promote mental and cording to clinical standards while effectively addressing physical health when faced with a myriad of significant complex patient needs that are often not directly ad- life issues impacting patients’ well-being. Financial strug- dressed in the EBP procedures. A critical finding of the gles, as one of our providers pointed out, is one example current study suggests that the therapeutic relationship, of the numerous life issues that may have an impact on individual patient factors, and system-level factors such patients’ physical and mental health (e.g., Brenk-Franz et as time and scheduling availability are critical drivers al. ). The impact of these issues was conceptualized that guide provider delivery and adaptations to ensure in the development of this brief CBT  and cannot be that EBPs “work” in clinical settings. disentangled from treating depression and anxiety. Pro- Prior literature documents provider modifications to EBP viders responded to this challenge in diverse ways; how- protocols implemented in community or specialty mental ever, common to all was the desire to enhance and health settings [35–37] and other interventions in the pri- maintain patient rapport while delivering an EBP during mary care setting ; however, this is the first study known this pragmatic trial. to the authors to report adaptations made in response to Study findings have potential implications for future challenges experienced when implementing an EBP in the implementation research, as well as policy and planning primary care setting. The DAP model was used to related to broader dissemination of EBPs to frontline contextualize emergent clinician qualitative themes into ad providers. For example, future research about imple- hoc adaptation categories detailed by the model, namely, menting psychotherapy in primary care would benefit provider skills and abilities, patient-emergent issues, and from the knowledge that the therapeutic relationship available resources. Prior research investigating adaptations and individual patient factors were critical drivers guid- during implementation in primary care of interventions de- ing how providers adapted the treatment protocol to signed to improve health promotion similarly found adapta- make it “work.” Moreover, dissemination of manualized tions based on patient preferences and circumstances . EBPs requires understanding and addressing the need Two DAP model dimensions, provider knowledge and for providers to modify treatment content and process organizational changes, were not identified as sources of to meet the unique needs of patients. Chambers and adaptation during the implementation of this brief CBT. Norton  argue that researchers should expect and More research is needed to determine what, if any, impact collect information on treatment adaptations made dur- these sources of adaptation have on implementation of psy- ing implementation and monitor these in terms of im- chotherapy in primary care. Alternative reasons may ac- pact on individuals, organizations, and communities. count for why these areas were not identified. For example, The amount of protocol modification is debatable and the interview guide did not elicit related provider experi- requires further data and exploration—especially as ences, or perhaps providers delivering the treatment may modification relates to patient outcomes. Although im- not be ideal participants for recognizing and reporting these plementation strategies such as fidelity monitoring and sources of adaptation. provider feedback can be used to reduce treatment Providers highlighted challenges they faced in their “drift,” it is important to acknowledge that fidelity mea- efforts to implement brief CBT in their practice. They sures typically do not provide an exhaustive list of all noted the importance of acknowledging or integrating prohibited and acceptable behaviors; therefore, fidelity patients’ broader life issues in the context of a fo- monitoring alone is insufficient for understanding if and cused treatment protocol and accommodating varia- how a modification impacts patient outcomes [41, 42]. tions in patient engagement with treatment. Also, Applied to the current study, for example, even though providers described challenges of learning how to providers achieved high fidelity ratings, treatment out- pace brief CBT sessions and treat a complex patient comes may have been even more successful if clinicians population in the primary care context. Facilitative ef- had not made any ad hoc adaptations. In fact, a recent forts were designed to ensure core treatment compo- meta-analysis found that treatment outcomes were unre- nents were delivered while also being flexible to lated to provider EBP adherence or competence . adaptations to accommodate provider styles and clin- Time was found to be a key resource driving adapta- ical needs as they arose. Use of audit and feedback of tions, in terms of best using the scheduled time and in session audio recordings was well-received by a rescheduling patients who miss treatment sessions. Des- multi-professional group of providers and appeared to pite early work to design a treatment protocol that successfully promote the balance between fidelity and works within the demanding primary care environment, flexibility while implementing this treatment. contextual issues such as scheduling and the manner in The therapeutic relationship is likely a key factor guid- which providers responded to patient no-shows affected ing when and how providers adapt a treatment protocol their reported ease of adopting the intervention into Mignogna et al. Implementation Science (2018) 13:76 Page 9 of 11 routine clinical practice. While advanced scheduling of providers’ feedback to make additional modifications to our treatment sessions at site 1 promoted weekly delivery of implementation strategy in real-time. Our study is also lim- brief CBT, normal interruptions to treatment occurred, ited in that we did not query other key stakeholders in the for example, patients not showing for an appointment. implementation process, such as clinic directors who could Given that all patients in this study had a cardiopulmo- have provided additional insights on the primary care set- nary condition, it was common for some missed ap- ting and identified possible unknown barriers and facilita- pointments to result from increased physical distress. tors to implementation. While the DAP model served as a Providers were challenged to deliver treatment with useful framework for the current study, one limitation of these missed appointments. The importance of time as a the model is its inability to clearly articulate overlap be- resource may be underappreciated in implementation re- tween its ad hoc adaptation categories. Qualitative interview search. Future efforts to implement brief CBT in pri- data were not always clearly sorted into only one category, mary care may benefit from working with program and several adaptations likely involve an interaction be- leadership to anticipate and advise providers as to how tween the provider’s style and the needs of the patient. An- to respond to missed appointments (e.g., ensuring pro- other limitation in the current study is that the reliability of viders have open slots in their clinic schedule to accom- the fidelity assessment method was not evaluated. Future modate patients’ needs to reschedule appointments in a research should look to design study methods that include a busy primary care environment). reliability assessment of the method used to monitor pro- Although the literature supports the effectiveness of vider fidelity to the treatment being implemented. phone-based delivery of EBPs, as well as its ability to reduce barriers and promote engagement in treatment [44, 45], Conclusions providers in this study voiced some challenges it brings to Providers’ perspectives reveal important insights into chal- delivering brief CBT in primary care. For example, some lenges of implementing EBPs in a busy primary care set- expressed how phone-delivered sessions led to some pa- ting and highlight tensions between treatment fidelity and tients allowing “other things to take precedence” over their ad hoc adaptations made to individualized treatment to treatment. In the current trial, patients were offered treat- address patient needs. Providers’ experiences during our ment in person or by phone. Empirical knowledge is needed implementation trial highlight potential adaptations to to guide provider decisions around how to implement a brief CBT that may serve a key role in its successful im- treatment with this degree of flexibility in treatment delivery plementation in the primary care setting. Future efforts to to promote high-quality care and address clinically relevant implement EBPs should recognize that adaptations are ex- concerns. Given demands for more efficient care delivery, pected occurrences during treatment delivery and attempt additional research is needed to understand the relationship to quantify and categorize adaptations in terms of impact between treatment modality and treatment fidelity. on delivery and, more importantly, on patient outcomes. Although our study is limited specifically to the primary care mental health integrated setting within the VHA, we believe our findings are transferable to other contexts (i.e., Additional files non-VHA settings) where primary care and mental/behav- ioral health services and providers are integrated. Findings Additional file 1: Protocol Adherence and Competency Evaluation (ACE) rating system for rating system for ACCESS (ACE-ACCESS). (PDF 367 kb) are also transferable more generally to settings where Additional file 2: Clinician Exit Interview Guide. (PDF 166 kb) evidence-based practices are implemented by frontline pro- viders who may experience the tension that arises between adaptation and fidelity when delivering these interventions Abbreviations in a real-world clinical context. Our work is limited in that ACE: Adherence and competence evaluation; CBT: Brief cognitive behavioral therapy; DAP: Dynamic Adaptation Process; EBPs: Evidence-based we conducted a summative rather than formative evaluation psychotherapies; PARIHS: Promoting Action on Research Implementation in  of providers’ perspectives on fidelity to the brief CBT Health Services; RE-AIM: Reach, Effectiveness, Adoption, Implementation, and intervention. Collecting one-time interviews at the end of Maintenance; VHA: Veterans Health Administration the study most likely limited the scope of providers’ re- sponses, given the length of time (e.g., many providers deliv- Acknowledgements ered care for over 1 year before interview data were The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs, the US collected) they were involved in the project. Important de- Government or Baylor College of Medicine, or other academic affiliates. tails may have been inadvertently omitted that would have Earlier versions of this work were presented at the 8th Annual Conference been captured if we had conducted a formative evaluation on the Science of Dissemination and Implementation, the 18th Annual VA Psychology Leadership Conference, and the 36th Society of Behavioral and captured providers’ viewpoints throughout the phases Medicine Annual Meeting. of our study. Formative-evaluation feedback would have The authors would like to thank Ms. Sonora Hudson for her thoughtful also augmented facilitation efforts by allowing us to use review and editing of this manuscript. Mignogna et al. Implementation Science (2018) 13:76 Page 10 of 11 Funding 4. McHugh RK, Barlow DH. The dissemination and implementation of The research reported here was supported by the Department of Veterans Affairs, evidence-based psychological treatments. A review of current efforts. Am Veterans Health Administration, and Health Services Research and Development Psychol. 2010;65:73–84. Service (IIR 09-088, PI Cully). Dr. Cully is an investigator in the Behavioral Health & 5. Meredith LS, Mendel P, Pearson M, Wu SY, Joyce G, Straus JB, et al. Implementation Program at the Michael E. DeBakey VA Medical Center Health Implementation and maintenance of quality improvement for treating Services Research and Development Center of Innovations, Houston, Texas. It was depression in primary care. Psychiatr Serv. 2006;57(1):48–55. also partly supported by the Department of Veterans Affairs, Veterans Health 6. McHugo GJ, Drake RE, Whitley R, Bond GR, Campbell K, Rapp CA, et al. Administration, Office of Research and Development, the Center for Innovations Fidelity outcomes in the National Implementing Evidence-Based Practices in Quality, Effectiveness and Safety (CIN 13-413), and the South Central Mental Project. Psychiatr Serv. 2007;58:1279–84. Illness, Research, Education, and Clinical Center. 7. Kauth MR, Sullivan G, Blevins D, Cully JA, Landes RD, Said Q, et al. Employing external facilitation to implement cognitive behavioral therapy Availability of data and materials in VA clinics: a pilot study. Implement Sci. 2010;5:75. The data generated by participant interviews during this study are not 8. Funderburk JS, Sugarman DE, Labbe AK, Rodrigues A, Maiso SA, Nelson B. publicly available because the data contain potentially identifying Behavioral health interventions being implemented in a VA primary care information, and participants were assured during the consent process that system. J Clin Psychol Med Settings. 2011;18:22–9. the information they provided would be publicly available only in aggregate. 9. Kearney LK, Post EP, Pomerantz AS, Zeiss AM. Aplying the interprofessional patient aligned care team in the Department of Veterans Affairs: Authors’ contributions transforming primary care. Am Psychol. 2014;69:399–408. JM, LM, JH, NH, MK, AD, KS, and JC made substantial contributions to the study 10. Nieuwsma JA, Trivedi RB, McDuffie J, Kronish I, Benjamin D, Williams JW. conception and design. JH conducted the interviews. LM and JM led the Brief psychotherapy for depression in primary care: a systematic review of qualitative analysis and interpretation of the interview data, developed and the evidence (VA-ESP Project #09-010). Washington, DC: Department of continually revised the codebook, and wrote the first draft of the manuscript. Veterans Affairs, Veterans Health Administration; 2011. JH and JC reviewed and refined the codebook. JB interpreted the data and 11. Cape J, Whittington C, Buszewicz M, Wallace P, Underwood L. Brief made critical revisions to the manuscript. All authors were involved in critically psychological therapies for anxiety and depression in primary care: meta- evaluating the manuscript and read and approved the final manuscript. analysis and meta-regression. BMC Med. 2010;8:38. 12. Cuijpers P, van Straten A, van Schaik A, Andersson G. Psychological Ethics approval and consent to participate treatment of depression in primary care: a meta-analysis. B J Gen Pract. Providers gave written consent to participate in the study and subsequently 2009;59:e51–60. verbally assented to the qualitative interview. Providers were contacted by 13. Waltz J, Addis ME, Koerner K, Jacobson NS. Testing the integrity of a email to schedule interviews at a convenient time. All study procedures were psychotherapy protocol: assessment of adherence and competence. J approved by each site’s respective Institutional Review Board and VHA Consult Clin Psychol. 1993;61:620. Research & Development Committee. 14. Eke AN, Neumann MS, Wilkes AL, Jones PL. Preparing effective behavioral interventions to be used by prevention providers: the role of Competing interests researchers during HIV prevention research trials. AIDS Educ Prev. 2006; The authors declare that they have no competing interests. 18(4 Suppl A):44–58. 15. Aarons GA, Green AE, Palinkas LA, Self-Brown S, Whitaker DJ, Lutzker JR, et al. Dynamic adaptation process to implement an evidence-based child Publisher’sNote maltreatment intervention. Implement Sci. 2012;7(1):32. Springer Nature remains neutral with regard to jurisdictional claims in 16. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability published maps and institutional affiliations. framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. Author details 17. Pérez D, Van der Stuyft P, Zabala MC, Castro M, Lefèvre P. A modified VISN 17 Center of Excellence for Research on Returning War Veterans, theoretical framework to assess implementation fidelity of adaptive Central Texas VA Health Care System, 4800 Memorial Drive (151C), Waco, TX public health interventions. Implement Sci. 2016;11:91. https://doi.org/ 76711, USA. Department of Psychiatry and Behavioral Sciences, Texas A&M 10.1186/s13012-016-0457-8. University Health Science Center, Temple, TX, USA. Houston VA Health 18. Cully JA,Armento ME, Mott J,Nadorff MR,NaikAD, Stanley MA, et al.Brief Services Research and Development Center of Innovations in Quality, cognitive behavioral therapy in primary care: a hybrid type 2 patient-randomized Effectiveness, and Safety, Michael E. DeBakey VA Medical Center, Houston, effectiveness-implementation design. Implement Sci. 2012;71(1):64. TX, USA. Section of Health Services Research, Department of Medicine, 19. Cully JA, Paukert A, Falco J, Stanley MA. Cognitive-behavioral therapy: Baylor College of Medicine, Houston, TX, USA. South Central Mental Illness innovations for cardiopulmonary patients with depression and anxiety. Research, Education and Clinical Center (SC MIRECC), Houston, TX, USA. Cogn Behav Pract. 2009;16:394–407. National Center for Posttraumatic Stress Disorder, Executive Division, White 20. Cully JA, Stanley MA, Deswal A, Hanania N, Phillips LL, Kunik ME. Cognitive- River Junction, VT, USA. Department of Psychiatry and Behavioral Sciences, behavioral therapy for chronic cardiopulmonary conditions: preliminary outcomes Baylor College of Medicine, Houston, TX, USA. Oklahoma City VA Health from an open trial. Prim Care Companion J Clin Psychiatry. 2010;12(4):e1–6. Care System, Oklahoma City, OK, USA. University of Oklahoma Health 21. Cully JA, Stanley MA, Petersen NJ, Hundt NE, Kauth MR, Naik AD, et al. Sciences Center, Oklahoma City, OK, USA. Department of Psychiatry, Dell Delivery of brief cognitive behavioral therapy for medically ill patients in Medical School, University of Texas, Austin, TX, USA. primary care: a pragmatic randomized clinical trial. J Gen Intern Med. 2017; [ Epub ahead of print] Received: 29 September 2017 Accepted: 21 May 2018 22. Hundt NE, Renn BN, Sansgiry S, Petersen NJ, Stanley MA, Kauth MR, Naik AD, Kunik ME, Cully J.A. (in press). Predictors of response to brief CBT in patients with cardiopulmonary conditions. Health Psychol. References 23. Mott JM, Hundt NE, Sansgiry S, Mignogna J, Cully JA. Changes in 1. Veterans Health Administration. Uniform mental health services in VA Psychotherapy Utilization Among Veterans With Depression, Anxiety, and medical centers and clinics (VHA handbook 1160.010). Washington, DC: PTSD. Psychiatr Serv. 2014;65(1):106–12. Department of Veterans Affairs; 2008. 2. Wenzel A, Brown GK, Karlin BE. Cognitive behavioral therpay for depression 24. Cully JA, Stanley MA, Kauth MR, Naik A, Kinik ME. ACCESS: adjusting to in veterans and military service members: therapist manual. Washington, chronic conditions with education, support, and skills. South Central (Rural) DC: Department of Veterans Affairs; 2011. MIRECC Clinical Education Products. 2012; Available at https://www.mirecc. 3. Karlin BE, Ruzek JL, Chard KM, Eftekhari A, Monson CM, Hembree EA, et al. va.gov/visn16/. Accessed 20 Aug 2017. Dissemination of evidence-based psychological treatments for 25. Mignogna J, Hundt NE, Kauth MR, Kunik ME, Sorocco KH, Naik AD, et al. posttraumatic stress disorder in the Veterans Health Administration. J Implementing brief cognitive behavioral therapy in primary care: a pilot Trauma Stress. 2010;23:663–73. study. Transl Behav Med. 2014;4:175–83. Mignogna et al. Implementation Science (2018) 13:76 Page 11 of 11 26. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Quality in Health Care. 2007;19:349–57. 27. Kitson A, Rycroft-Maline J, Harvey G, McCormack B, Seers K, Titchen A. Evaluating the successful implementation of evidence into practice using the PARIHS framework: theoretical and practical challenges. Implement Sci. 2008;3(1):1. 28. Stetler CB, Damschroeder LJ, Helfrich CE, Hagedorn HJ. A guide for applying a revised version of the PARIHS framework for implementation. Implement Sci. 2011;6:99. 29. RE-AIM. http://www.re-aim.org/. Accessed 20 Aug 2017. 30. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7. 31. Birken SA, Powell BJ, Shea CM, Haines ER, Kirk MA, Leeman J, Rohweder C, Damschroder L, Presseau J. Criteria for selecting implementation science theories and frameworks: results from an international survey. Implement Scik. 2017;12:124. https://doi.org/10.1186/s13012-017-0656-y. 32. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–88. 33. Ryan B. Analyzing qualitative data. Systematic approaches. Thousand Oaks: Sage; 2010. 34. Kendall PC, Gosch E, Furr JM, Sood E. Flexibility within fidelity. J Am Acad Child Adolesc Psychiatry. 2008;47:987–93. 35. Stirman SW, Gutner C, Crits-Christoph P, Edmunds J, Evans AC, Beidas RS. Relationships between clinician-level attributes and fidelity-consistent and fidelity-inconsistent modifications to an evidence-based psychotherapy. Implement Sci. 2015;10(1):115. 36. Cook JM, Dinnen S, Thompson R, Simiola V, Schnurr PP. Changes in implementation of two evidence-based psychotherapies for PTSD in VA residential treatment programs: a national investigation. J Trauma Stress. 2014;27:137–43. 37. Stirman SW, Calloway A, Toder K, Miller CJ, De Vito AK, Meisel SN, et al. Modifications to cognitive therapy by community mental health providers: implications for effectiveness and sustainability. Psychiatr Serv. 2013;54:10. 38. Cohen DJ, Crabtree BF, Etz RS, Balasubramanian BA, Donahue KE, Leviton LC, Clark EC, Isaacson NF, Strange KC, Green LW. Fidelity versus flexibility. Am J Prevent Med. 2008;35:S381–9. 39. Brenk-Franz K, Strauβ B, Atiesler F, Fleischer C, Schneider N, Gensichen J. Patient-provider relationship as mediator between adult attachment and self-management in primary care patients with multiple chronic conditions. J Psychosom Res. 2017;97:131–5. 40. Chambers DA, Norton WE. The adaptome; advancing the science of intervention adaptation. Am J Prev Med. 2016;51(452):S124–31. 41. Stirman SW, Miller CJ, Toder K, calloway P. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65. 42. Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17. 43. Webb CA, Derubeis RJ, Barber JP. Therapist adherence/competence and treatment outcome: a meta-analytic review. J Consult Clin Psychol. 2010;78:200–11. 44. Egede LE, Acierno R, Knapp RG, Walker RJ, Payne EH, Frueh BC. Psychotherapy for depression in older veterans via telemedicine: effect on quality of life, satisfaction, treatment credibility, and service delivery perception. J Clin Psychiatry. 2016;77:1704–11. 45. Mohr DC, He J, Duffecy J, Reifler D, Sokol L, Burns MN, et al. Effect of telephone-administered vs face-to-face cognitive behavioral therapy on adherence to therapy and depression outcomes among primary care patients: a randomized trial. JAMA. 2012;307i:2278–85. 46. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21(Suppl 2):S1–8.
– Springer Journals
Published: Jun 5, 2018