Abstract State health departments commonly use quality improvement coaching as an implementation strategy for improving low human papillomavirus (HPV) vaccination coverage, but such coaching can be resource intensive. To explore opportunities for improving efficiency, we compared in-person and webinar delivery of coaching sessions on implementation outcomes, including reach, acceptability, and delivery cost. In 2015, we randomly assigned 148 high-volume primary care clinics in Illinois, Michigan, and Washington State to receive either in-person or webinar coaching. Coaching sessions lasted about 1 hr and used our Immunization Report Card to facilitate assessment and feedback. Clinics served over 213,000 patients ages 11–17. We used provider surveys and delivery cost assessment to collect implementation data. This report is focused exclusively on the implementation aspects of the intervention. More providers attended in-person than webinar coaching sessions (mean 9 vs. 5 providers per clinic, respectively, p = .004). More providers shared the Immunization Report Card at clinic staff meetings in the in-person than webinar arm (49% vs. 20%; p = .029). In both arms, providers’ belief that their clinics’ HPV vaccination coverage was too low increased, as did their self-efficacy to help their clinics improve (p < .05). Providers rated coaching sessions in the two arms equally highly on acceptability. Delivery cost per clinic was $733 for in-person coaching versus $461 for webinar coaching. In-person and webinar coaching were well received and yielded improvements in provider beliefs and self-efficacy regarding HPV vaccine quality improvement. In summary, in-person coaching cost more than webinar coaching per clinic reached, but reached more providers. Further implementation research is needed to understand how and for whom webinar coaching may be appropriate. Implications Practice: Public health practitioners who deliver immunization quality improvement programs may be able to reach more providers through in-person coaching than webinar coaching. Policy: Public health organizations that sponsor immunization quality improvement programs should allocate funding for costs incurred by in-person delivery. Research: Future research should identify ways for increasing the reach of webinar coaching so as to improve the effectiveness of this highly acceptable, low-cost alternative to in-person coaching. BACKGROUND Human papillomavirus (HPV) vaccine is licensed to prevent several HPV-related cancers . Routine vaccination at age 11 or 12 years has been recommended by the Advisory Committee on Immunization Practices (ACIP) since 2006 for females and since 2011 for males . In 2016, ACIP updated HPV vaccine recommendations to include a two-dose schedule for adolescents initiating the vaccination series before their 15th birthday (0, 6–12 month schedule) . Three doses remain recommended for those who initiate the vaccination series at age 15 or older (0, 1–2, 6 month schedule) . Despite effectiveness and national recommendations for routine administration of HPV vaccine, many age-eligible adolescents in the USA are not vaccinated according to schedule. As of 2016, less than two thirds (60%) of 13- to 17-year olds in the USA had initiated HPV vaccination . This coverage is considerably lower than for other vaccines in the adolescent platform: tetanus, diphtheria, and pertussis (Tdap; 88%) and meningococcal conjugate (82%) . In the USA, state policies have contributed to high rates of Tdap and meningitis vaccination as most states require these vaccines for school entry . However, only Rhode Island, Virginia, and Washington DC have implemented such policies for HPV vaccination . Political concerns, including more limited support for HPV vaccine school-entry requirements among parents  and healthcare providers , have been barriers to the enactment of these laws. Thus, successful delivery of HPV vaccine depends to a large extent on healthcare providers’ recommendation practices. A key reason that adolescents are not up-to-date with HPV vaccination is that many healthcare providers do not consistently recommend that adolescents complete the series by age 13 [9,10]. Provider-level barriers to HPV vaccination underscore the need for immunization quality improvement (QI) interventions that can decrease missed opportunities in primary care clinics . One evidence-based approach to immunization QI is the Centers for Disease Control and Prevention’s (CDC) Assessment, Feedback, Incentives and eXchange (AFIX) program . AFIX uses strategies, including assessment and feedback, in the context of brief, in-person coaching sessions. In these sessions, an immunization QI coach from a state or regional health department evaluates a clinic’s vaccination coverage levels and delivers training to clinic providers on immunization best practices. Based on data that show that AFIX increases early childhood vaccination coverage , the CDC recommends that each year health departments deliver AFIX to at least one quarter of providers of federally funded vaccines . The AFIX program’s existing infrastructure and success raise the possibility that HPV vaccine-focused AFIX coaching could increase coverage among adolescents. In a pilot trial, we conducted with 91 primary care clinics in North Carolina and found that, at 5-month follow-up, AFIX coaching modestly improved HPV vaccination coverage among adolescents, ages 11–12, compared to control clinics . Although many health departments have begun using a modified version of AFIX to address the underuse of adolescent vaccines, including HPV vaccine, the use of AFIX to improve adolescent vaccination is understudied. Rigorous evaluation of an HPV vaccine-focused AFIX, in states with a range of capacity to deliver immunization QI coaching, is needed to inform national practice. To address this gap, we developed HPV vaccine-focused QI coaching tools, in partnership with immunization specialists in health departments in Illinois, Michigan, and Washington State. These tools included suggested scripts for the AFIX coaches who deliver the QI sessions, as well as materials such as an Immunization Report Card and an Action Plan (materials are available online at www.hpviq.org). In 2015, our three health department partners incorporated the tools into their routine adolescent AFIX programs. Together, we conducted a randomized controlled trial with 224 primary care clinics to evaluate the effectiveness of HPV vaccine-focused QI coaching (or “HPV AFIX”) for raising HPV vaccination coverage; clinics received the intervention via traditional, in-person coaching or interactive webinar. We found that in-person coaching achieved modest, but consistent improvements across time (6- and 12-month postintervention) and states for HPV vaccine initiation (≥1 dose) among adolescent ages 11–12, compared to no-coaching control (Gilkey, Calo, Leeman et al., in preparation). Webinar coaching also achieved statistically significant improvements but these were smaller than in-person coaching and inconsistent across time points and states. To better understand how delivery mode may affect the implementation of HPV AFIX, we conducted a process evaluation that compared in-person coaching sessions to those delivered via interactive webinar in terms of reach, acceptability, perceptions about HPV vaccine QI, adoption of recommended QI strategies, barriers to implementing QI strategies, and delivery cost. METHODS Clinic recruitment and assignment to trial arm The trial included primary care clinics specializing in pediatric and family medicine. To ensure high reach, we included clinics having at least 500 patients, ages 11–17, with active records in the Illinois, Michigan, and Washington immunization information systems (IIS). In March 2015, we randomized these 925 clinics, using a 1:1:1 ratio within state, to receive either in-person HPV AFIX coaching, webinar HPV AFIX coaching, or no coaching (control). Recruitment to the study started with the first clinic in each list of study arms and alternated between arms in order to conduct in-person and webinar coaching over the same time period. In total, 76 clinics received in-person coaching, 72 clinics received webinar coaching, and 76 clinics were in the control arm. Study clinics served over 312,000 patients, ages 11–17, with active records in the IIS. This report examines the 148 clinics, with 213,006 age-eligible patients, assigned to the in-person and webinar intervention arms. We do not report further on clinics in the control arm because we did not collect implementation data from these clinics. Fifty-four percent of clinics contacted to participate in the in-person arm enrolled in the trial (21 clinics were ineligible, 20 did not respond, 24 declined or cancelled), and 49% of clinics contacted to participate in the webinar arm enrolled (17 clinics were ineligible, 37 did not respond, 22 declined or cancelled). This report is focused exclusively on the implementation aspects of the intervention. Intervention From April to October 2015, immunization QI coaches from the state health departments delivered one in-person or webinar HPV AFIX coaching session to each intervention clinic. Prior to each session, the QI coach used the state IIS to generate a one-page Report Card showing the clinic’s current vaccination coverage levels for three vaccines routinely delivered to adolescents: HPV, meningococcal, and Tdap vaccines. Coaching consisted of a single session, designed to be ~60 min in length, in which a QI coach met with providers (i.e., physicians, nurse practitioners, clinic managers, other vaccine providers, and clinic staff) to discuss the clinic’s immunization coverage levels. The QI coach encouraged the attendance of all of the clinic’s providers and staff, offering up to 1 hr of continuing medical education (CME) credit as an incentive for participation. The QI coach began each session by sharing the clinic’s immunization coverage assessment. Specifically, the QI coach used the Report Card to show providers how their clinic’s HPV vaccination coverage compared to their Tdap and meningococcal vaccination coverage levels, which are typically higher than HPV vaccination. This assessment gave providers a baseline they could use to measure the success of their immunization QI activities. The QI coach then provided information about immunization best practices, including strategies to decrease missed opportunities, and evaluated the clinic’s potential areas for improvement from a list of 18 immunization best practices developed by the CDC . The QI coach also discussed HPV vaccination guidelines and the importance of the vaccine providers’ role in making a strong recommendation for HPV vaccination. The QI coach asked providers to set a goal to improve their HPV vaccination coverage by delivering the first dose of HPV vaccine to at least 10% of their clinic’s 11- to 12-year-old patients over the next 6 months. The QI coach then worked with providers to identify one or two QI strategies from a one-page Action Plan to address challenges specific to their clinic to reach the HPV vaccination goal. At 3 and 6 months after the session, the QI coach sent the clinic an updated Report Card so that providers could assess their progress. These follow-up reports, along with a library of email coaching materials that we also developed with our state partners, were designed to support QI coaches to stay in touch with providers and maintain their engagement. In-person and webinar coaching were designed to deliver the same content. Webinar coaching used an interactive software platform, such as GoToMeeting, that allowed the QI coach to interact with providers in real time, similar to in-person coaching. The software included a screen-sharing function that allowed the QI coach to show materials to coaching session attendees. QI coaches used a structured slide set with talking points for their webinars and most also used them for the in-person coaching. The trial was registered in clinicaltrials.gov (NCT02370459) and was ruled exempt by the University of North Carolina Institutional Review Board. Implementation data collection procedure We collected data concurrent with the implementation of HPV vaccine QI coaching from provider surveys and delivery cost assessment. Provider surveys Providers at each clinic completed three online surveys: a precoaching survey prior to the coaching session, a postcoaching survey directly after the coaching session, and a follow-up survey 6 months after receiving coaching. Although we encouraged the same person to respond all surveys, not all clinics had the same provider completing the three surveys. Response rates were 88%, 85%, and 62%, respectively. Completing the postcoaching survey allowed providers to claim CME credit. Coaches sent up to five reminders, by email and phone, to nonrespondents. Prior to fielding the surveys, we cognitively tested our instruments with two physicians and two nurses. Surveys are available online at www.hpviq.org. Delivery cost To assess the costs that state health departments incurred delivering HPV AFIX coaching, we examined weekly time logs, salary data, and nonstaffing expenses. State partners logged their time spent on intervention activities (e.g., Report Card preparation, travel, QI session delivery) via a weekly online survey throughout a 58-week trial period. Compliance for completion of the weekly time logs was 100%, with state partners reporting 16% of time logs retrospectively 1–4 weeks after being due. We assessed current salary data, including fringe benefits, for all state partners who participated in the study using publicly available state employee salary information or requesting these data from state departments of human resources. Nonstaffing expenses (i.e., mileage reimbursement, lodging, air travel, meals, and webinar-hosting fees) came from purchase orders and invoice records kept by the research coordinator using an expense tracking log. We excluded costs incurred for research purposes (e.g., sending survey reminders). Because we were interested in assessing HPV AFIX coaching costs to state health departments, we did not collect costs incurred by providers, patients, or parents. Measures Reach We recorded the number and type of providers who attended coaching sessions, as well as those who claimed CME credit. We also assessed with whom providers shared the clinic’s Report Card and how they shared it via the follow-up survey (i.e., one-on-one conversations, announcements at a staff meeting, via email, or displaying the report in the office). Acceptability The postcoaching survey assessed overall satisfaction with items that measured the extent to which attendees found QI coaching sessions easy to understand, convenient, helpful, and well-facilitated. The 5-point response scale ranged from “strongly disagree” to “strongly agree.” The postcoaching survey also assessed the importance of individual session components, including review of the clinic’s adolescent vaccination coverage levels, discussion of provider recommendation of HPV vaccine, goal setting to improve HPV vaccination coverage, and selection of immunization QI strategies. The 5-point response scale ranged from “not important” to “extremely important.” A final post-coaching survey item assessed whether providers would, if given the choice, have preferred receiving the coaching via in-person or webinar delivery. Beliefs and self-efficacy Pre- and post-coaching surveys assessed beliefs that HPV vaccination QI was an important goal, that HPV vaccination coverage is low, that the clinic can improve HPV vaccination coverage, and that the provider can help lead those efforts (self-efficacy). The surveys also assessed providers’ belief that the immunization QI coaching and tools improved HPV vaccination coverage in their clinics. The 5-point response scale ranged from “strongly disagree” to “strongly agree.” Adoption of QI strategies The follow-up survey assessed which QI strategies, if any, clinics tried, as a result of the coaching session, to improve their HPV vaccine coverage. Strategies were as follows: encouraging providers to routinely recommend vaccination, reviewing CDC guidelines with staff, training front office staff on how to schedule vaccination appointments, signing standing orders, using reminder/recall systems, and providing educational materials to parents and patients. Barriers to implementing QI strategies The follow-up survey assessed the biggest barrier clinics faced to improving HPV vaccination coverage: staff turnover, limited staff time to implement QI activities, lack of support from leadership, limited stock of HPV vaccine, and time constraints during patient visits. Analyses We compared study arms on clinic characteristics using chi-square tests and analyses of variance. We used t tests to compare clinics in the in-person and webinar arms on mean scores related to acceptability, beliefs, and self-efficacy. These analyses accounted for provider clustering by clinic and state using the svy command . Statistical tests were two-tailed with a critical alpha of .05. We also compared in-person and webinar arms within each state, adjusting for clinic level clustering of providers; we used a critical alpha of .01 to correct for multiple comparisons. After applying this correction, we did not observe significant differences between study arms within states (results not shown). We analyzed survey data using Stata 13.0 (College Station, TX). To assess the cost to state health departments of delivering HPV vaccine QI coaching sessions, we used micro costing methods. This approach involves separately identifying, measuring, and valuing the resources utilized in the intervention . Using 2015 and 2016 salaries, we calculated personnel costs by multiplying the time spent on each intervention activity by the adjusted salary per hour of the staff. We calculated intervention cost per clinic by dividing the total intervention cost for each study arm by the number of clinics in that arm. We calculated intervention cost per provider by dividing the total intervention cost for each study arm by the average number of providers per clinic who attended the QI coaching session. We analyzed cost data using Excel. RESULTS Overall, 165 providers completed the precoaching survey, 229 completed the postcoaching survey, and 102 completed the 6-month follow-up survey (Table 1). Across the three surveys, the majority of providers practiced in family medicine clinics (57%–68%). Most respondents were nurses (31%–46%) or clinic managers (18%–32%); physicians constituted a smaller proportion (5%–10%) of survey respondents. Many respondents had worked in their current role for ten or more years (39%–56%). Over half (61%) of those who participated in the coaching sessions and responded to the evaluation survey claimed the CME credit we offered, including 78% of participating physicians (14/18) and 65% of nurses (68/104). Claiming CME credit did not vary by delivery mode. Table 1 | Respondent characteristics Surveys Pre-coaching Post-coaching Follow-up n = 165 n = 229 n = 102 n (%) n (%) n (%) Practice location Illinois 52 (32) 58 (25) 45 (44) Michigan 37 (22) 74 (32) 21 (21) Washington 76 (46) 97 (42) 36 (35) Practice specialty Pediatrics 59 (36) 73 (32) 44 (43) Family medicine 106 (64) 156 (68) 57 (57) Role Physicians 9 (5) 18 (8) 10 (10) Nurses 56 (34) 104 (46) 32 (31) Other vaccine providers 15 (9) 20 (9) 5 (5) Clinic managers 47 (28) 42 (18) 33 (32) Other staff roles 3 (2) 1 (<1) 0 (0) Other staff 35 (21) 43 (19) 22 (22) Years in role <5 65 (39) 75 (33) 19 (19) 5–9 36 (22) 55 (24) 26 (25) ≥10 64 (39) 99 (43) 57 (56) Claimed CME credit n/a 140 (61) n/a Surveys Pre-coaching Post-coaching Follow-up n = 165 n = 229 n = 102 n (%) n (%) n (%) Practice location Illinois 52 (32) 58 (25) 45 (44) Michigan 37 (22) 74 (32) 21 (21) Washington 76 (46) 97 (42) 36 (35) Practice specialty Pediatrics 59 (36) 73 (32) 44 (43) Family medicine 106 (64) 156 (68) 57 (57) Role Physicians 9 (5) 18 (8) 10 (10) Nurses 56 (34) 104 (46) 32 (31) Other vaccine providers 15 (9) 20 (9) 5 (5) Clinic managers 47 (28) 42 (18) 33 (32) Other staff roles 3 (2) 1 (<1) 0 (0) Other staff 35 (21) 43 (19) 22 (22) Years in role <5 65 (39) 75 (33) 19 (19) 5–9 36 (22) 55 (24) 26 (25) ≥10 64 (39) 99 (43) 57 (56) Claimed CME credit n/a 140 (61) n/a Response rates for precoaching, postcoaching, and follow-up surveys were 88%, 85%, and 62%, respectively (response defined as at least 1 respondent per clinic). n/a not applicable. View Large Table 1 | Respondent characteristics Surveys Pre-coaching Post-coaching Follow-up n = 165 n = 229 n = 102 n (%) n (%) n (%) Practice location Illinois 52 (32) 58 (25) 45 (44) Michigan 37 (22) 74 (32) 21 (21) Washington 76 (46) 97 (42) 36 (35) Practice specialty Pediatrics 59 (36) 73 (32) 44 (43) Family medicine 106 (64) 156 (68) 57 (57) Role Physicians 9 (5) 18 (8) 10 (10) Nurses 56 (34) 104 (46) 32 (31) Other vaccine providers 15 (9) 20 (9) 5 (5) Clinic managers 47 (28) 42 (18) 33 (32) Other staff roles 3 (2) 1 (<1) 0 (0) Other staff 35 (21) 43 (19) 22 (22) Years in role <5 65 (39) 75 (33) 19 (19) 5–9 36 (22) 55 (24) 26 (25) ≥10 64 (39) 99 (43) 57 (56) Claimed CME credit n/a 140 (61) n/a Surveys Pre-coaching Post-coaching Follow-up n = 165 n = 229 n = 102 n (%) n (%) n (%) Practice location Illinois 52 (32) 58 (25) 45 (44) Michigan 37 (22) 74 (32) 21 (21) Washington 76 (46) 97 (42) 36 (35) Practice specialty Pediatrics 59 (36) 73 (32) 44 (43) Family medicine 106 (64) 156 (68) 57 (57) Role Physicians 9 (5) 18 (8) 10 (10) Nurses 56 (34) 104 (46) 32 (31) Other vaccine providers 15 (9) 20 (9) 5 (5) Clinic managers 47 (28) 42 (18) 33 (32) Other staff roles 3 (2) 1 (<1) 0 (0) Other staff 35 (21) 43 (19) 22 (22) Years in role <5 65 (39) 75 (33) 19 (19) 5–9 36 (22) 55 (24) 26 (25) ≥10 64 (39) 99 (43) 57 (56) Claimed CME credit n/a 140 (61) n/a Response rates for precoaching, postcoaching, and follow-up surveys were 88%, 85%, and 62%, respectively (response defined as at least 1 respondent per clinic). n/a not applicable. View Large Reach In-person HPV coaching reached more providers than webinar coaching. An average of nine providers (range: 1–33) attended in-person sessions, whereas five providers (range: 1–26) attended webinar sessions (p = .004). The majority of providers (87%) reported sharing the Report Card with others in their clinics. They reported sharing the Report Card with vaccine providers (69%), clinic managers (52%), or other providers (17%). Report Card sharing occurred through in-person, one-on-one conversations (63%), announcements at a clinic’s staff meeting (39%), email communications (18%), or displaying the report in the clinic (10%). More providers in the in-person coaching arm shared the Report Card at a clinic’s staff meeting compared with those in the webinar arm (49% vs. 20%, respectively; p = .029). No other differences between arms were observed for sharing the Report Card. Impact on beliefs, self-efficacy, and immunization QI strategies Both in-person and webinar coaching improved providers’ beliefs about HPV vaccine QI and self-efficacy. More specifically, pre- and post-coaching comparisons demonstrated that coaching increased providers’ beliefs that their clinic’s HPV vaccination coverage was low, their belief that the clinic could improve coverage, and their self-efficacy to help lead those QI efforts (all pre/postcomparisons p < .05; Fig. 1). At 6-month follow-up, over half of respondents reported believing that implementing the QI activities improved their clinic’s HPV vaccine coverage levels (54% somewhat or strongly agreed). We did not find evidence to suggest that beliefs and self-efficacy differed across delivery modes. Fig 1 View largeDownload slide | Changes in providers’ beliefs and self-efficacy about immunization QI before and after receiving an HPV QI coaching session. Error bars show standard errors. Fig 1 View largeDownload slide | Changes in providers’ beliefs and self-efficacy about immunization QI before and after receiving an HPV QI coaching session. Error bars show standard errors. Providers in the in-person and webinar arms reported similarly on clinics’ implementation of immunization QI activities. At 6-month follow-up, the majority of clinics reported having conducted efforts to encourage providers to routinely recommend HPV vaccine (95%), provide educational materials to parents and patients (84%), and review CDC guidelines for HPV vaccination with staff (66%). Fewer than half of the clinics trained front office staff on how to schedule appointments (49%), used reminder/recall systems (47%), or signed standing orders (34%) for HPV vaccination. Most providers listed limited staff time (31%), time constraints during patient visits (33%), or staff turnover (19%) as the biggest barrier to implementing QI activities. Few providers reported lack of support from leadership (7%) or limited stock of HPV vaccine (5%) as the major barrier. Perceived barriers did not differ by arm. Acceptability Providers rated both in-person and webinar HPV QI coaching and tools highly on acceptability (Table 2). These measures included ease of understanding (mean = 4.5), convenience (mean = 4.3), helpfulness (mean = 4.1), and facilitation (mean = 4.4). Providers also rated highly the importance of individual intervention components such as using the Report Card to review clinic’s immunization coverage rates (mean = 4.3), discussing provider recommendation of HPV vaccine (mean = 4.3), setting a 6-month goal to improve HPV vaccination levels (mean = 3.9), and selecting immunization QI strategies (mean = 4.1). Providers in the in-person and webinar arms gave the same high scores on these measures of satisfaction and importance. In terms of intervention delivery mode, slightly more providers (p = .01) who received in-person coaching preferred this mode (85%), compared to those who received and preferred webinar coaching (79%). Table 2 | Mean (SD) acceptability scores by coaching delivery mode In-person Webinar Overalla Ease of understanding 4.5 (1.0) 4.6 (1.0) Convenience 4.2 (1.1) 4.4 (1.0) Helpfulness 4.0 (1.1) 4.3 (0.9) Facilitation 4.3 (1.2) 4.5 (1.1) Specific componentsb Review immunization rates 4.3 (0.8) 4.3 (0.8) Discuss provider recommendation 4.3 (0.8) 4.4 (0.7) Set 6-month goal 4.0 (1.0) 3.8 (0.8) Select quality improvement strategies 4.1 (0.9) 4.1 (0.8) In-person Webinar Overalla Ease of understanding 4.5 (1.0) 4.6 (1.0) Convenience 4.2 (1.1) 4.4 (1.0) Helpfulness 4.0 (1.1) 4.3 (0.9) Facilitation 4.3 (1.2) 4.5 (1.1) Specific componentsb Review immunization rates 4.3 (0.8) 4.3 (0.8) Discuss provider recommendation 4.3 (0.8) 4.4 (0.7) Set 6-month goal 4.0 (1.0) 3.8 (0.8) Select quality improvement strategies 4.1 (0.9) 4.1 (0.8) In-person and webinar arms had comparable acceptability scores (all comparisons p > .05). Comparisons adjusted for clustering by state and clinic. aItems assessed with a 5-point response scale ranging from “strongly disagree” to “strongly agree.” bItems assessed with a 5-point response scale ranging from “not important” to “extremely important.” View Large Table 2 | Mean (SD) acceptability scores by coaching delivery mode In-person Webinar Overalla Ease of understanding 4.5 (1.0) 4.6 (1.0) Convenience 4.2 (1.1) 4.4 (1.0) Helpfulness 4.0 (1.1) 4.3 (0.9) Facilitation 4.3 (1.2) 4.5 (1.1) Specific componentsb Review immunization rates 4.3 (0.8) 4.3 (0.8) Discuss provider recommendation 4.3 (0.8) 4.4 (0.7) Set 6-month goal 4.0 (1.0) 3.8 (0.8) Select quality improvement strategies 4.1 (0.9) 4.1 (0.8) In-person Webinar Overalla Ease of understanding 4.5 (1.0) 4.6 (1.0) Convenience 4.2 (1.1) 4.4 (1.0) Helpfulness 4.0 (1.1) 4.3 (0.9) Facilitation 4.3 (1.2) 4.5 (1.1) Specific componentsb Review immunization rates 4.3 (0.8) 4.3 (0.8) Discuss provider recommendation 4.3 (0.8) 4.4 (0.7) Set 6-month goal 4.0 (1.0) 3.8 (0.8) Select quality improvement strategies 4.1 (0.9) 4.1 (0.8) In-person and webinar arms had comparable acceptability scores (all comparisons p > .05). Comparisons adjusted for clustering by state and clinic. aItems assessed with a 5-point response scale ranging from “strongly disagree” to “strongly agree.” bItems assessed with a 5-point response scale ranging from “not important” to “extremely important.” View Large Delivery costs The time spent per clinic to deliver the full intervention was 12.6 hr in the in-person study arm, including 1.1 hr to conduct the QI session, and 9.0 hr in the webinar arm, including 0.9 hr to conduct the QI session (Table 3). In the in-person arm, staff time traveling to coaching sessions was the most time-consuming activity (3.6 hr/clinic). In both arms, the most time consuming of the remaining activities were follow-up with clinics (3.3 hr/clinic), session preparation (2.7–2.8 hr/clinic), and recruitment and scheduling (2.0 hr/clinic). Table 3 | Average staff time spent and cost per HPV AFIX coaching session Time spent Delivery cost In-person hours (%) Webinar hours (%) In-person $ (%) Webinar $ (%) Staffing Clinic recruitment and scheduling 2.0 (16) 2.0 (22) 117 (16) 117 (25) Session preparation 2.7 (21) 2.8 (31) 129 (18) 136 (30) Coaching session 1.1 (9) 0.9 (10) 65 (9) 56 (12) Travel to/from session 3.6 (29) n/a 233 (32) n/a Follow-up with clinics 3.3 (26) 3.3 (37) 141 (19) 141 (31) Travel Mileage and meals n/a n/a 20 (3) n/a Lodging and airfare n/a n/a 28 (4) n/a Webinar license n/a n/a n/a 11 (2) Total 12.6 9.0 733 461 Time spent Delivery cost In-person hours (%) Webinar hours (%) In-person $ (%) Webinar $ (%) Staffing Clinic recruitment and scheduling 2.0 (16) 2.0 (22) 117 (16) 117 (25) Session preparation 2.7 (21) 2.8 (31) 129 (18) 136 (30) Coaching session 1.1 (9) 0.9 (10) 65 (9) 56 (12) Travel to/from session 3.6 (29) n/a 233 (32) n/a Follow-up with clinics 3.3 (26) 3.3 (37) 141 (19) 141 (31) Travel Mileage and meals n/a n/a 20 (3) n/a Lodging and airfare n/a n/a 28 (4) n/a Webinar license n/a n/a n/a 11 (2) Total 12.6 9.0 733 461 n/a not applicable. View Large Table 3 | Average staff time spent and cost per HPV AFIX coaching session Time spent Delivery cost In-person hours (%) Webinar hours (%) In-person $ (%) Webinar $ (%) Staffing Clinic recruitment and scheduling 2.0 (16) 2.0 (22) 117 (16) 117 (25) Session preparation 2.7 (21) 2.8 (31) 129 (18) 136 (30) Coaching session 1.1 (9) 0.9 (10) 65 (9) 56 (12) Travel to/from session 3.6 (29) n/a 233 (32) n/a Follow-up with clinics 3.3 (26) 3.3 (37) 141 (19) 141 (31) Travel Mileage and meals n/a n/a 20 (3) n/a Lodging and airfare n/a n/a 28 (4) n/a Webinar license n/a n/a n/a 11 (2) Total 12.6 9.0 733 461 Time spent Delivery cost In-person hours (%) Webinar hours (%) In-person $ (%) Webinar $ (%) Staffing Clinic recruitment and scheduling 2.0 (16) 2.0 (22) 117 (16) 117 (25) Session preparation 2.7 (21) 2.8 (31) 129 (18) 136 (30) Coaching session 1.1 (9) 0.9 (10) 65 (9) 56 (12) Travel to/from session 3.6 (29) n/a 233 (32) n/a Follow-up with clinics 3.3 (26) 3.3 (37) 141 (19) 141 (31) Travel Mileage and meals n/a n/a 20 (3) n/a Lodging and airfare n/a n/a 28 (4) n/a Webinar license n/a n/a n/a 11 (2) Total 12.6 9.0 733 461 n/a not applicable. View Large The intervention cost per clinic was much higher for in-person than webinar coaching ($733 vs. $461), but intervention cost per provider was lower for in-person than webinar coaching ($81 vs. $92, respectively). In the in-person arm, staff time traveling accounted for almost one third (32%; $233 per clinic) of the cost. Recruitment and scheduling accounted $117 per clinic, and state personnel averaged four attempts to reach and schedule clinics, which did not differ by arm. The per clinic cost associated with session preparation and follow-up was $129 (18%) and $141 (19%), respectively, for in-person coaching, and $136 (30%) and $141 (31%), respectively, for webinar coaching. Mileage, meals, lodging, and airfare accounted for 7% ($48 per clinic) of the in-person arm cost, and webinar licensing comprised 2% ($11 per clinic) of the webinar-delivered cost. DISCUSSION Using implementation data from a large, multistate trial with primary care clinics, this process evaluation compared in-person HPV vaccine QI coaching sessions to those delivered via interactive webinar, finding relative advantages of each. In-person delivery reached more providers and staff as more attended coaching sessions and more shared the Report Card at their clinics when compared to webinar delivery. Providers rated both arms highly in terms of acceptability and showed similar, positive changes in provider beliefs about key immunization QI activities. Webinar coaching, however, cost about one third less per clinic than in-person coaching, although it cost more per provider in the coaching session. Thus, state health departments seeking to incorporate webinar-delivered coaching to their AFIX programs must weigh the cost-saving advantage of webinars against more limited reach, particularly given that the webinar coaching produced small, inconsistent improvements in HPV vaccination coverage. Overall, clinic participation was high across study arms. By using CME incentives and making a concerted effort to reach multiple providers per clinic, we were able to draw seven providers to QI coaching sessions on average, compared to typically one provider per clinic in our pilot trial . However, in-person delivery outperformed webinar delivery on reach in the current trial with nine versus five providers participating on average, respectively. In-person participants were also more likely to share the Report Card in staff meetings. Differences in provider participation may be explained by several factors. First, in the webinar arm, providers may have had limited access to resources (e.g., multiple computers, conference space) needed to join the webinar without disrupting workflow or to accommodate a larger number of providers at the same time. On the other hand, in-person delivery may have elicited a more collegial environment or seemed more in line with usual norms for attending face-to-face staff meetings, resulting in higher participation . Receiving printed copies of the Report Card, which may have occurred with in-person but not webinar delivery, may have similarly facilitated dissemination of this document to colleagues. Whatever the case, to increase reach, future efforts may wish to explore the use of recorded webinars as a way to extend reach, so that providers who are not able to join the in-person session may still benefit from the content shared. Six months after participating in the coaching session, providers reported engaging in a variety of immunization QI activities. Almost every provider in both arms reported encouraging other clinic providers to routinely recommend HPV vaccine. A central component of the intervention was to discuss the importance of provider recommendation, and our Report Card offered providers examples on how to make a strong recommendation for HPV vaccination. This QI strategy was prioritized in our intervention as over 70% of adolescents who receive a recommendation initiate HPV vaccination . However, implementation of other practice-based approaches was modest with, for example, less than half of providers reporting that their clinics used reminder/recall systems or standing orders. These two approaches are associated with increased immunization coverage among young children and are recommended by the Community Preventive Services Task Force . Use of reminder/recall systems, in particular, was found to improve HPV vaccine uptake in a recent systematic review . Despite the evidence, it was not a surprise to find that reminder/recall systems were not widely implemented as these systems require staff resources that not all clinics can support . Furthermore, half of our sample reported that the main barriers for implementing QI activities were lack of staff time and high staff turnover. One approach to overcome these barriers is the use of a collaborative centralized reminder/recall system between health departments and clinics. A recent randomized trial with 576 primary care practices showed that a collaborative centralized reminder/recall system was more effective and more cost-effective than a practice-based system . In addition, most providers reported they preferred that the health department, rather than their practice, conduct reminder/recall notifications centrally , showing this approach is also highly accepted among providers. The two arms achieved very similar results in terms of acceptability and improving providers’ perceptions about HPV vaccine QI efforts. This finding is in line with our pilot intervention  and prior studies  showing satisfaction levels with online-delivered training as high as in-person delivery. Although webinar can introduce some challenges to QI coaches and providers in terms of how they interact with each other, providers rated coaches highly in the way they facilitated sessions. Our intervention materials provided coaches with structured techniques to engage providers throughout the session (e.g., a didactic PowerPoint presentation that guided discussions between QI coaches and providers and contained prompts to facilitate conversations) so, whatever the delivery mode, the session was designed to build rapport and facilitate two-way discussions. Both arms also achieved statistically significant improvements in provider beliefs and self-efficacy to help raise clinic’s HPV vaccine coverage. Taken together, these findings suggest that webinar delivery could substantially increase providers’ positive perceptions towards engaging in QI activities without adversely affecting their satisfaction with QI coaching, although data on effectiveness are needed. Per clinic, we found that webinar coaching cost a third less than in-person coaching due to reduced travel costs. However, per provider, webinar coaching cost slightly more compared to in-person coaching due to higher provider participation in the in-person arm. Webinar coaching could be a lower-cost delivery mode if state immunization QI programs increase the number of providers who attend this type of sessions and webinars raise vaccination coverage. The lower costs of webinar coaching could extend the reach of immunization QI programs in many ways. In our trial, for example, we could have delivered HPV vaccine QI coaching to an additional 45 primary care clinics had we used the funds spent on in-person coaching for webinar coaching instead. Webinar delivery also eliminates other key challenge state health department personnel faces when delivering in-person coaching sessions, including the amount of time they spend traveling to clinics. Travel accounted for almost one third (29%) of the time reported and most of the time and cost difference between the two delivery modes. In addition, in states with large geographic areas, like Illinois, Michigan, and Washington, travel can pose a major inconvenience, so clinics in rural or isolated areas may be especially well-served by webinar delivery. Even though webinar delivery took less time, it is important to note that both delivery modes took a substantial commitment in terms of recruitment and scheduling of clinics. Delivering email coaching (i.e., 3- and 6-month follow-up Report Cards) also took a lot of time, and might be made more efficient through the use of automated email systems . Trial vaccination coverage analyses are pending; webinar would need to be as effective as in-person coaching for the cost-savings to have practical value to programs. In terms of strengths, our trial employed a strong research design, including random allocation of clinics and a large sample of clinics. We also worked with a range of state health departments, so we obtained implementation data that could resemble the capacities to deliver immunization QI coaching in many states. In our trial, contrary to our pilot in which one person delivered all coaching sessions, several QI coaches delivered in-person and webinar coaching, which improves the generalizability of our findings. Limitations include the use of self-reported measures to assess process measures; future studies should consider using fidelity monitoring to assess intervention delivery (whether or not QI coaches delivered coaching sessions as intended) and adoption (whether or not providers participated and implemented QI activities as reported). Our micro costing method focused on intervention delivery so as to compare delivery modes; we excluded factors (e.g., overhead expenses, provider costs) that would be needed to calculate overall program costs or to conduct a cost-effectiveness analysis. Additional research is needed to understand how our findings generalize to clinics with lower patient volumes, particularly given that small clinics may have limited resources to implement immunization QI activities or reduced capacity to support webinar delivery. We focused on medium-to-large clinics as the CDC prioritized them for receiving AFIX because they can have a larger impact on vaccination rates . Lastly, future research should explore the relationship between implementation processes (e.g., reach, acceptability, perceptions about HPV vaccine QI, barriers to implementing QI strategies) and vaccination rates. CONCLUSIONS The CDC is currently investing resources in strengthening AFIX for HPV vaccination, and our trial implementation findings may inform these efforts to maximize the effective use and integration of QI coaching strategies into existing state immunization programs. Webinar technology could be a promising tool for delivering the AFIX model, but more research is needed to overcome the more limited reach of this approach and to understand the impact of webinar coaching on vaccination coverage. State health departments should focus on how to overcome the shortcomings of this method, while capitalizing on the cost and time savings and broad acceptability of this approach to HPV vaccine QI efforts. Forthcoming effectiveness data will be important for understanding how and for whom webinar delivery of HPV vaccine QI coaching and tools is appropriate. Acknowledgements: We thank our state partners for their support and collaboration: Susan Williams and Linda Kasebier from the Illinois Department of Public Health; Rachel Potter from the Michigan Department of Community Health; and Nicole Freeto from the Washington State Department of Health. Funding: The Robert Wood Johnson Foundation provided a research grant that funded the trial (grant no. 71272). Authors’ time was supported by training and career development grants from the National Cancer Institute: R25CA116339 and K22CA186979. Funders played no role in (a) study design; (b) the collection, analysis, and interpretation of data; (c) the writing of the report; or (d) the decision to submit the manuscript for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute or the Robert Wood Johnson Foundation. Compliance with Ethical Standards Primary Data: The findings reported have not been previously published, and this manuscript is not being simultaneously submitted elsewhere. The authors presented some preliminary data from this work at the 37th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine. The authors have full control of all primary data and agree to allow the journal to review the data if requested. Informed Consent: The trial was ruled exempt by the University of North Carolina Institutional Review Board, including the collection of informed consent from participants. Statement of Human Rights: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Statement on the Welfare of Animals: This article does not contain any studies with animals performed by any of the authors. Conflict of Interest: Brewer has received commercial research grants from Merck and Pfizer and is a consultant/advisory board member for Merck. He is chair of the National HPV Vaccination Roundtable. The other authors have no financial disclosures or potential conflicts of interest to report. References 1. Viens LJ , Henley SJ , Watson M et al. Human Papillomavirus-Associated Cancers – United States, 2008-2012 . MMWR Morb Mortal Wkly Rep . 2016 ; 65 ( 26 ): 661 – 666 . Google Scholar CrossRef Search ADS PubMed 2. Petrosky E , Bocchini JA Jr , Hariri S et al. ; Centers for Disease Control and Prevention (CDC) . Use of 9-valent human papillomavirus (HPV) vaccine: updated HPV vaccination recommendations of the advisory committee on immunization practices . MMWR Morb Mortal Wkly Rep . 2015 ; 64 ( 11 ): 300 – 304 . Google Scholar PubMed 3. Meites E , Kempe A , Markowitz LE . Use of a 2-dose schedule for human papillomavirus vaccination – updated recommendations of the advisory committee on immunization practices . MMWR Morb Mortal Wkly Rep . 2016 ; 65 ( 49 ): 1405 – 1408 . Google Scholar CrossRef Search ADS PubMed 4. Walker TY , Elam-Evans LD , Singleton JA et al. National, regional, state, and selected local area vaccination coverage among adolescents aged 13–17 years – United States, 2016 . MMWR Morb Mortal Wkly Rep . 2017 ; 66 ( 33 ): 874 – 882 . Google Scholar CrossRef Search ADS PubMed 5. Bugenske E , Stokley S , Kennedy A , Dorell C . Middle school vaccination requirements and adolescent vaccination coverage . Pediatrics . 2012 ; 129 ( 6 ): 1056 – 1063 . Google Scholar CrossRef Search ADS PubMed 6. National Conference of State Legislatures . HPV Vaccine Policies . Washington, DC : National Conference of State Legislatures . 2016 [cited September 16, 2015]. Available at http://www.ncsl.org/research/health/hpv-vaccinestate-legislation-and-statutes.aspx. 7. Calo WA , Gilkey MB , Shah PD , Moss JL , Brewer NT . Parents’ support for school-entry requirements for human papillomavirus vaccination: a national study . Cancer Epidemiol Biomark Prev . 2016 ; 25 ( 9 ): 1317 – 1325 . Google Scholar CrossRef Search ADS 8. Califano S , Calo WA , Weinberger M , Gilkey MB , Brewer NT . Physician support of HPV vaccination school-entry requirements . Hum Vaccin Immunother . 2016 ; 12 ( 6 ): 1626 – 1632 . Google Scholar CrossRef Search ADS PubMed 9. Darden PM , Thompson DM , Roberts JR et al. Reasons for not vaccinating adolescents: National Immunization Survey of Teens, 2008–2010 . Pediatrics . 2013 ; 131 ( 4 ): 645 – 651 . Google Scholar CrossRef Search ADS PubMed 10. Gilkey MB , Calo WA , Moss JL , Shah PD , Marciniak MW , Brewer NT . Provider communication and HPV vaccination: the impact of recommendation quality . Vaccine . 2016 ; 34 ( 9 ): 1187 – 1192 . Google Scholar CrossRef Search ADS PubMed 11. President’s Cancer Panel Annual Report . Accelerating HPV Vaccine Uptake: Urgency for Action to Prevent Cancer. A Report to the President of the United States from the President’s Cancer Panel . Bethesda, MD : National Cancer Institute ; 2014 . 12. Centers for Disease Control and Prevention . AFIX (Assessment, Feedback, Incentives, and eXchange) . Available at http://www.cdc.gov/vaccines/programs/afix/index.html. Accessibility verified April 3, 2017 . 13. Jacob V , Chattopadhyay SK , Hopkins DP , Murphy Morgan J , Pitan AA , Clymer JM ; Community Preventive Services Task Force . Increasing coverage of appropriate vaccinations: a community guide systematic economic review . Am J Prev Med . 2016 ; 50 ( 6 ): 797 – 808 . Google Scholar CrossRef Search ADS PubMed 14. Gilkey MB , Dayton AM , Moss JL et al. Increasing provision of adolescent vaccines in primary care: a randomized controlled trial . Pediatrics . 2014 ; 134 ( 2 ): e346 – e353 . Google Scholar CrossRef Search ADS PubMed 15. Centers for Disease Control and Prevention . AFIX (Assessment, Feedback, Incentives, and eXchange) . AFIX Site Visit Answer Guide. Available at https://www.cdc.gov/vaccines/programs/afix/site-visit-answers.html. Accessibility verified April 3, 2017 . 16. Desai M , Begg MD . A comparison of regression approaches for analyzing clustered data . Am J Public Health . 2008 ; 98 ( 8 ): 1425 – 1429 . Google Scholar CrossRef Search ADS PubMed 17. Drummond MF , Sculpher MJ , Claxton K , Stoddart GL , Torrance GW. Methods for the Economic Evaluation of Health Care Programmes . 3rd ed . New York, NY : Oxford University Press ; 2005 : 71 – 72 . 18. Gilkey MB , Moss JL , Roberts AJ , Dayton AM , Grimshaw AH , Brewer NT . Comparing in-person and webinar delivery of an immunization quality improvement program: a process evaluation of the adolescent AFIX trial . Implement Sci . 2014 ; 9 : 21 . Google Scholar CrossRef Search ADS PubMed 19. Avila CC , Quinn VP , Geiger AM , Kerby TJ , St Charles M , Clough-Gorr KM . Webinar training: an acceptable, feasible and effective approach for multi-site medical record abstraction: the BOWII experience . BMC Res Notes . 2011 ; 4 : 430 . Google Scholar CrossRef Search ADS PubMed 20. Dorell CG , Yankey D , Santibanez TA , Markowitz LE . Human papillomavirus vaccination series initiation and completion, 2008–2009 . Pediatrics . 2011 ; 128 ( 5 ): 830 – 839 . Google Scholar CrossRef Search ADS PubMed 21. Niccolai LM , Hansen CE . Practice- and community-based interventions to increase human papillomavirus vaccine coverage: a systematic review . JAMA Pediatr . 2015 ; 169 ( 7 ): 686 – 692 . Google Scholar CrossRef Search ADS PubMed 22. Pereira JA , Quach S , Heidebrecht CL et al. ; Public Health Agency of Canada/Canadian Institutes of Health Research Influenza Research Network (PCIRN) Vaccine Coverage Theme Group . Barriers to the use of reminder/recall interventions for immunizations: a systematic review . BMC Med Inform Decis Mak . 2012 ; 12 : 145 . Google Scholar CrossRef Search ADS PubMed 23. Kempe A , Saville AW , Dickinson LM et al. Collaborative centralized reminder/recall notification to increase immunization rates among young children: a comparative effectiveness trial . JAMA Pediatr . 2015 ; 169 ( 4 ): 365 – 373 . Google Scholar CrossRef Search ADS PubMed 24. Albright K , Saville A , Lockhart S , Widmer Racich K , Beaty B , Kempe A . Provider attitudes toward public-private collaboration to improve immunization reminder/recall: a mixed-methods study . Acad Pediatr . 2014 ; 14 ( 1 ): 62 – 70 . Google Scholar CrossRef Search ADS PubMed 25. Chiswell M , Smissen A , Ugalde A et al. Using webinars for the education of health professionals and people affected by cancer: processes and evaluation . J Canc Educ . In press. doi:10.1007/s13187-016-1138-7 © The Society of Behavioral Medicine 2018. All rights reserved. For permissions, please e-mail: email@example.com
Translational Behavioral Medicine – Oxford University Press
Published: Feb 20, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera