Development of a web-based toolkit to support improvement of care coordination in primary care

Development of a web-based toolkit to support improvement of care coordination in primary care Abstract Promising practices for the coordination of chronic care exist, but how to select and share these practices to support quality improvement within a healthcare system is uncertain. This study describes an approach for selecting high-quality tools for an online care coordination toolkit to be used in Veterans Health Administration (VA) primary care practices. We evaluated tools in three steps: (1) an initial screening to identify tools relevant to care coordination in VA primary care, (2) a two-clinician expert review process assessing tool characteristics (e.g. frequency of problem addressed, linkage to patients’ experience of care, effect on practice workflow, and sustainability with existing resources) and assigning each tool a summary rating, and (3) semi-structured interviews with VA patients and frontline clinicians and staff. Of 300 potentially relevant tools identified by searching online resources, 65, 38, and 18 remained after steps one, two and three, respectively. The 18 tools cover five topics: managing referrals to specialty care, medication management, patient after-visit summary, patient activation materials, agenda setting, patient pre-visit packet, and provider contact information for patients. The final toolkit provides access to the 18 tools, as well as detailed information about tools’ expected benefits, and resources required for tool implementation. Future care coordination efforts can benefit from systematically reviewing available tools to identify those that are high quality and relevant. Implications Practice: To narrow down the very large number of available care coordination tools to those relevant to local improvement goals and context, healthcare leaders and managers can adapt and apply the tool selection process we outline in this article. Policy: A structured approach to toolkit development can facilitate the spread of care innovations for quality improvement efforts, but toolkits are an adjunct to, not a replacement for, a serious commitment on the part of clinic leadership and staff to improve care coordination. Research: To ensure that toolkits are effective in achieving their desired ends, additional work should characterize the development processes and design features of different toolkits, including the relationship between these processes and features and subsequent tool adoption, implementation, and sustainability. INTRODUCTION Inefficiencies in coordination of chronic care are a significant source of waste in the United States (USD $25–$45 billion in 2011) [1] and a barrier to achieving the Triple Aim of improved patient experience of care, lower per capita costs, and better population health [2]. In the Veterans Health Administration (VA), recent legislation allowing some patients to seek care from community medical providers (i.e. outside of VA owned and operated facilities) has further complicated care coordination in the primary care practices of a system where patients already have higher rates of comorbidity and mental illness than private sector counterparts [3, 4]. Although a systematic review has shown the benefits of care coordination strategies in the published research literature [5], effective methods to identify, document, spread, and sustain evidence-based care coordination approaches in routine care are not clear. The Coordination Toolkit and Coaching (CTAC) project aims to improve patients’ experience of care coordination within VA primary care and between VA primary care and other outpatient settings, including health care provided in the community, while also developing better methods for bringing research evidence on care coordination into routine care. To implement research on care coordination in practice, any coordination strategy must not only be evidence-based, but also applicable to the context of the adopting organization. The gap between research and practice exists in part because research results for complex problems such as care coordination are often presented generically, without the supporting documentation of the tools required to achieve such results. Additionally, research results often do not apply when conducted under the varied resource constraints, patient populations, and organizational milieus that are found in practice [6]. These constraints may strongly shape the specific intervention elements that can be applied successfully within any specific care coordination improvement effort. For example, though the VA’s shared information technology infrastructure greatly facilitates internal sharing of innovations, it also means that innovations in research studies or available in general repositories such as the Agency for Healthcare Research and Quality (AHRQ) Innovations Exchange may not be applicable to a local VA initiative because of incompatibility with VA’s electronic health record. In addition, individuals delivering clinical care may have limited time to search for tools, make a comparison between available options, and decide which tools to adopt. Thus, one way to facilitate the uptake of research in practice is to compile a toolkit—in essence, a “short list” of action-oriented strategies for improving care coordination that is compatible with the local context [7]. In this article, we describe CTAC’s first phase, which involved selecting tools for a toolkit and developing a VA Intranet site to support the tools. We aimed to evaluate candidate tools based on the existing evidence base for effective care coordination strategies, relevance to improving patients’ experience of care coordination, tool feasibility and quality, and evaluations by VA patients and frontline VA primary care providers and staff. Within the VA, toolkits are often used in conjunction with other dissemination methods, such as learning collaboratives [8] and quality improvement coaching, and a later phase of CTAC will compare the additional value of distance-based coaching to implement the tools, with access to the toolkit without coaching. METHODS Definitions Care coordination is defined as “…the deliberate organization of patient care activities between two or more participants (including the patient) involved in a patient’s care to facilitate the appropriate delivery of health care services. Organizing care involves the marshalling of personnel and other resources needed to carry out all required patient care activities, and is often managed by the exchange of information among participants responsible for different aspects of care” [9]. A toolkit is defined as “a collection of related information, resources, or tools that together can guide users to develop a plan or organize efforts to follow evidence-based recommendations or meet evidence-based specific practice standards,” and a tool as “an instrument (e.g. survey, guidelines, or checklist) that helps users accomplish a specific task that contributes to meeting a specific evidence-based recommendation or practice standard” [10]. Organizational context The VA is a national integrated healthcare delivery system serving 5.9 million Veteran patients annually [11], with a central headquarters in Washington, DC and 18 healthcare networks known as Veterans Integrated Service Networks (VISNs); each VISN includes multiple healthcare “facilities” that are systems in themselves, offering inpatient and outpatient care, often across multiple sites. In the VA, most patients are assigned to a primary care team [12]. Since 2010, the VA has focused on transforming its primary care teams into a patient-centered medical home model with augmented staffing (one registered nurse care manager, one licensed practical nurse, and one administrative clerk for each primary care provider), called patient-aligned care teams (PACT), and PACTs are tasked with coordinating the patient’s care [13]. VA’s status as an integrated healthcare delivery system offers the benefits of a shared electronic health record and communication systems, including electronic mail, for VA-owned and operated facilities and selected contractors. Toolkit development The decision of which tools to include in the toolkit involved three steps: (1) an initial review of content to determine if tools met inclusion or exclusion criteria, (2) a two-clinician review process to rate tools on their characteristics, and (3) semi-structured interviews with VA patients and frontline clinicians and staff. Figure 1, adapted from the PRISMA statement [14], shows the flow of tools through this three-step process. Fig 1 View largeDownload slide Three-step tool identification, screening and eligibility determination process. 1Preliminary exclusion criteria included a tool being: (1) condition-specific, (2) covering only clinical preventive services, (3) intended for use outside of primary care, (4) not focused on care coordination, (5) not truly a tool, (6) VA already using the tool or strategy widely, (7) not applicable to the VA setting, (8) part of a toolkit and could not stand alone as an individual tool, or (9) not coachable by CTAC’s distance-based coach. 2Exclusions based on the tool-rating checklist (Supplementary Appendix) and reviewer discussion and consensus. 3Excluded tools classified as supporting materials or additional resources for the included tools. Fig 1 View largeDownload slide Three-step tool identification, screening and eligibility determination process. 1Preliminary exclusion criteria included a tool being: (1) condition-specific, (2) covering only clinical preventive services, (3) intended for use outside of primary care, (4) not focused on care coordination, (5) not truly a tool, (6) VA already using the tool or strategy widely, (7) not applicable to the VA setting, (8) part of a toolkit and could not stand alone as an individual tool, or (9) not coachable by CTAC’s distance-based coach. 2Exclusions based on the tool-rating checklist (Supplementary Appendix) and reviewer discussion and consensus. 3Excluded tools classified as supporting materials or additional resources for the included tools. In the first step, the principal investigator and project manager identified tools using a snowball approach, starting with two VA toolkits (focused on PACT and specialty care) [15, 16] and two nonVA toolkits [17, 18]. The focus of this initial review was to determine the candidate tools’ general relevance to the project topic. We excluded tools for any of the following reasons: tools were restricted to a particular clinical domain, such as being condition-specific (e.g. only for diabetes) or covering only clinical preventive services; were intended for use primarily in a setting outside of primary care (e.g. specialty care, inpatient care); were not focused on care coordination (e.g. only access focused, only continuity focused, or only general quality improvement); were not truly a tool (e.g. general information sheets that were not action-oriented); the VA was already disseminating the tool or strategy widely (e.g. tools to encourage uptake of secure messaging between patients and providers); the tool was not applicable to the VA setting; the tool was part of a toolkit with interdependent tools that could not stand alone as individual tools; or CTAC’s distance-based coach would not be able to help primary care teams with using the tool. The principal investigator, project manager, and two additional project staff then sorted remaining tools meeting inclusion criteria into categories and sub-categories to organize them according to the topics they covered. In the second step, two clinicians independently reviewed remaining tools, using a newly developed and piloted tool-rating checklist (Supplementary Appendix). During the second step, the principal investigator served as the first reviewer of all the tools, with the role of second reviewer split between five clinicians, who were assigned roughly equal numbers of tools to review. After the reviews were completed, the first and second reviewers met to discuss substantial differences in ratings (defined as a two-point difference on any 3-point rating of a given dimension of the tool, or at least a 3-point difference on the 7-point tool summary rating). After reviewers discussed these differences, they had the option of changing their ratings if desired. In the third step, the principal investigator (interviewer) and project manager (note-taker) conducted semi-structured interviews with VA providers, clinic staff, and patients in order to obtain feedback about the tools and learn more about how the tools might function in a clinic setting. A total of eight interviews were completed on a broad selection of the included tools, five of them individual interviews with PACT clinic staff from one VA site (primary care provider, registered nurse, lead clinic clerk, psychologist and pharmacist) and three of them group interviews from another site (two Veterans receiving health care services at the VA, members of the 21-member VISN 22 clinician Community of Practice, and six members of the nine-member VISN 22 Nursing Workgroup). Interviews lasted 1 hr each, and were all conducted by phone except for the in-person group interview of the two Veteran patients. A pre-determined selection of tools, specifically chosen for each interviewee based on his/her role at the VA, was then presented, via email attachment, an online screen-sharing program, or in person, as applicable. Interviewees provided feedback about the tools with interviewer prompting via questions such as “Do you have a similar tool in use at your facility?”, “Would a tool like this be helpful at your facility?”, and “How would this tool be used at your facility?” The group interviews of VA providers and staff involved a specific clinical scenario and set of tools that were presented to the group, with the group then providing feedback. After these three steps were completed, final assignment of tools to categories occurred, and tools were deployed to an online website accessible via the VA Intranet, using SharePoint®. Using rapid cycles of development, a SharePoint programmer worked with the principal investigator and project manager to develop the toolkit website’s functionality, usability, and look and feel. An internal SharePoint site was chosen because of rapidity of deployment and the ease with which tools could be added or changed in the future. We placed special emphasis on developing “tool access pages” (Supplementary Appendix), which provide prospective users with a preview of the amount of time and effort required to deploy the tool, and the expected benefits of the tool. Evaluation of toolkit development process To evaluate the process of tool selection and toolkit development, we conducted a review of documentation collected during the process, including spreadsheets of included and excluded tools, reviewer ratings of tools, and field notes from semi-structured interviews. We carried out quantitative analyses of reconciled clinician reviewer scores from the second step of the tool selection process, calculating median and interquartile ranges for tool summary ratings, which were on a 1–7 integer scale (a higher number being more favorable). For inferential statistics, we used the Wilcoxon signed-rank test to assess reviewer agreement on tool ratings, and the Wilcoxon rank-sum test for ratings of included versus excluded tools. Two-tailed p values were considered significant at p < .05. We also evaluated the degree to which the final set of tools selected for inclusion conformed to evidence-based strategies, and the dimensions of health care quality that the tools might affect. To determine the degree to which tools covered the range of evidence-based strategies for care coordination, we mapped the tools to a compilation of care coordination strategies shown effective in a systematic review [5]. We also mapped the pathways by which the selected tools might affect dimensions of the VA’s Strategic Analytics for Improvement and Learning (SAIL) domains, which are an internal dashboard used by VA leaders to benchmark VA facilities’ performance [19]. As the CTAC project focuses on improving patients’ experience of care coordination, we determined in more detail the mapping of tools to the Health System Hassles Scale [20], which assesses patients’ challenges using the healthcare system, such as getting medications refilled on time, lack of information about why patients were referred to a specialist, poor communication between different doctors or clinics, and disagreements between doctors about the patient’s diagnosis or treatment, and is included in the AHRQ Care Coordination Measures Atlas [9]. Ethics approval The CTAC project was determined to be nonresearch by the VA Office of Primary Care Services (in accordance with VHA Handbook 1058.05) [21] and by the VA Greater Los Angeles Healthcare System Institutional Review Board. RESULTS Review of 22 toolkits, manuals and resources (the original four toolkits plus 18 found through a snowball search) resulted in identification of 300 potential tools from these sources (Figure 1). Application of initial inclusion and exclusion criteria narrowed the list of potential tools to 65. The two-clinician review process and subsequent project team discussion further reduced the number of candidate tools to 38. These 38 tools had higher clinician ratings than excluded tools; however, the difference in summary ratings between included and excluded tools was only statistically significantly different for the first reviewer, who reviewed all the tools himself (reviewer 1: median ± interquartile range, 5 ± 1 for included tools vs. 4 ± 1 for excluded tools, p = .0005). For the five different clinicians who shared the role of second reviewer, the collective rating of included and excluded tools showed more variation around median values (reviewer 2: median ± interquartile range, 5 ± 3 for included tools versus 4 ± 2 for excluded tools, p = .096). There was no significant difference in summary ratings between the two reviewers of each tool (p = .915). Semi-structured interviews highlighted factors that might suggest a tool should be included or excluded. For example, one set of tools presented during an interview was focused on motivational interviewing techniques and supporting curricula and resources. The discussion during the interview revealed that training in this area and supplemental materials are already available to VA clinic staff. Other interviews revealed specific challenges a facility might face using a tool. For instance, when presented with an agenda-setting tool that patients receive at check-in and fill out in the waiting room, one interviewee described check-in kiosks used at the clinic, and explained that Veterans do not typically see a clerk if they check in via the kiosk. Therefore, having a clerk hand a paper-based agenda-setting form to the patient at check-in would not be a viable approach in this clinic setting. Based on the feedback from the semi-structured interviews, we were able to reduce the 38 tools at this stage to the final set of 18 (originally 19, with two tools combined into one; Table 1), with many of the excluded 19 tools available on the toolkit website as additional resources or supporting materials for the included tools. Table 1 Care coordination toolkit tool list with descriptions Tool category and tool name Tool description Managing Referrals to Specialty Care VA Service Agreement Template A template with the purpose of facilitating timely access and patient-centered care for patients by promoting an effective relationship between the PACT team and the specialty care team during the management of a patient’s care. Consult Guides for Primary Care (Information to Include) A comprehensive list of clinical conditions and for each, a standard set of information to be included in a referral request, to help ensure a meaningful visit when the referred patient visits the relevant specialist. While the tool consists of multiple data sets, not all need to be implemented. Relaxation and Meditation Program: An approach to self-management of mental health in primary care Information and tools for implementing a program that teaches patients relaxation and meditation skills, with the potential benefit of decreasing the number of referrals to specialty mental health services. Medication Management VA Formulary (Abbreviated) The Primary Care Service in the VA Palo Alto Healthcare System worked with its pharmacy department to create an abbreviated and searchable VA formulary. This resource was posted on the healthcare system’s external web page and is updated periodically for accuracy. AudioRENEWAL: Phone-based Medication Renewal for Patients Allows patients to request a renewal of their prescription directly from within the AudioCARE telephone refill system 24 hr a day, 7 days a week. Communication with Community Providers to Co-Manage Veterans’ Care A two-page document that includes a letter to help facilitate communication between a VA provider and an outside healthcare provider regarding management of the Veteran patient and instructions for obtaining medications through the VA, and a guide for providers (nonVA) to request a prescription medication that is nonformulary. Medication Tracker for Patients A one-page form with sections to be filled in by a member of the PACT team for the patient. Spaces are provided to fill in a patient’s medication details and healthcare provider name and contact information. The medication list includes details that can help patients remember when to take their medications and what dose to take. Patient After-Visit Summary After-Visit Instructions for patients (paper) A customizable document instructing patients about where to go before leaving the VA the day of their appointment, as well as any follow-up actions which need to be taken. Includes contact information for specialty clinics as well as a map of the campus, which can be used to direct patients to their next destination on campus. After-Visit Summary for patients (electronic) An electronic tool that produces a customizable, printable patient summary that can be provided to a patient after his/her primary care office visit to summarize visit content and subsequent action steps, if appropriate. Patient Activation Materials Patient Agenda-Setting Form A simple, easy to fill-out form to help patients prepare for their healthcare visit. The form can be sent to patients in the mail with a pre-visit packet, or can be given to patients to complete in the waiting room. Patient Treatment Decision Guide A worksheet for patients to use when faced with a medical decision. The form includes questions to ask the provider during the medical appointment, space to take notes, and guidance on how to proceed with making a decision. Tips for Patients: Improving Communication with your Primary Care Team A two-page handout that can be mailed to patients in a pre-visit packet or given to patients at check-in, before their appointment. The tool includes tips for communicating with the primary care team, questions for patients to think about before their appointment and space to write down answers. This tool can help ensure all of a patient’s questions and concerns are addressed during their medical appointment. Pharmacy Safety for Patients A patient handout that explains the role of the pharmacist, pharmacy and patient during the process of obtaining medications. Tips for Patients: Questions to Ask Before, During and After your Visit A guide that includes tips, ideas, and questions for patients to use before, during, and after their appointment that will help facilitate a successful healthcare visit. Provider Contact Information for Patients Clinic Information Pamphlet A template that clinics can customize to create a clinic information pamphlet for patients that includes pertinent information about the clinic. Information may include appointment-making instructions, provider contact information, prescription refill instructions, and other clinic details. My Primary Care Team: Contact and Other Information A one-page handout that lists important provider contact information for a patient. Includes contact details for the primary care provider, PACT nurse, tele-health nurse, specialty care, pharmacy, and mental health. My Primary Care Team: Wallet Card with Contact Information A customizable wallet-sized card that can be printed, populated with a patient’s primary care team information and given to the patient. The back of the card includes a list of information patients should provide when leaving a message at the clinic. Save a Trip to Primary Care A one-page resource for patients, explaining and illustrating the difference between routine, nonurgent, and urgent medical situations. The sheet explains to patients what actions to take in the case of each of these situations, and provides relevant contact information. Tool category and tool name Tool description Managing Referrals to Specialty Care VA Service Agreement Template A template with the purpose of facilitating timely access and patient-centered care for patients by promoting an effective relationship between the PACT team and the specialty care team during the management of a patient’s care. Consult Guides for Primary Care (Information to Include) A comprehensive list of clinical conditions and for each, a standard set of information to be included in a referral request, to help ensure a meaningful visit when the referred patient visits the relevant specialist. While the tool consists of multiple data sets, not all need to be implemented. Relaxation and Meditation Program: An approach to self-management of mental health in primary care Information and tools for implementing a program that teaches patients relaxation and meditation skills, with the potential benefit of decreasing the number of referrals to specialty mental health services. Medication Management VA Formulary (Abbreviated) The Primary Care Service in the VA Palo Alto Healthcare System worked with its pharmacy department to create an abbreviated and searchable VA formulary. This resource was posted on the healthcare system’s external web page and is updated periodically for accuracy. AudioRENEWAL: Phone-based Medication Renewal for Patients Allows patients to request a renewal of their prescription directly from within the AudioCARE telephone refill system 24 hr a day, 7 days a week. Communication with Community Providers to Co-Manage Veterans’ Care A two-page document that includes a letter to help facilitate communication between a VA provider and an outside healthcare provider regarding management of the Veteran patient and instructions for obtaining medications through the VA, and a guide for providers (nonVA) to request a prescription medication that is nonformulary. Medication Tracker for Patients A one-page form with sections to be filled in by a member of the PACT team for the patient. Spaces are provided to fill in a patient’s medication details and healthcare provider name and contact information. The medication list includes details that can help patients remember when to take their medications and what dose to take. Patient After-Visit Summary After-Visit Instructions for patients (paper) A customizable document instructing patients about where to go before leaving the VA the day of their appointment, as well as any follow-up actions which need to be taken. Includes contact information for specialty clinics as well as a map of the campus, which can be used to direct patients to their next destination on campus. After-Visit Summary for patients (electronic) An electronic tool that produces a customizable, printable patient summary that can be provided to a patient after his/her primary care office visit to summarize visit content and subsequent action steps, if appropriate. Patient Activation Materials Patient Agenda-Setting Form A simple, easy to fill-out form to help patients prepare for their healthcare visit. The form can be sent to patients in the mail with a pre-visit packet, or can be given to patients to complete in the waiting room. Patient Treatment Decision Guide A worksheet for patients to use when faced with a medical decision. The form includes questions to ask the provider during the medical appointment, space to take notes, and guidance on how to proceed with making a decision. Tips for Patients: Improving Communication with your Primary Care Team A two-page handout that can be mailed to patients in a pre-visit packet or given to patients at check-in, before their appointment. The tool includes tips for communicating with the primary care team, questions for patients to think about before their appointment and space to write down answers. This tool can help ensure all of a patient’s questions and concerns are addressed during their medical appointment. Pharmacy Safety for Patients A patient handout that explains the role of the pharmacist, pharmacy and patient during the process of obtaining medications. Tips for Patients: Questions to Ask Before, During and After your Visit A guide that includes tips, ideas, and questions for patients to use before, during, and after their appointment that will help facilitate a successful healthcare visit. Provider Contact Information for Patients Clinic Information Pamphlet A template that clinics can customize to create a clinic information pamphlet for patients that includes pertinent information about the clinic. Information may include appointment-making instructions, provider contact information, prescription refill instructions, and other clinic details. My Primary Care Team: Contact and Other Information A one-page handout that lists important provider contact information for a patient. Includes contact details for the primary care provider, PACT nurse, tele-health nurse, specialty care, pharmacy, and mental health. My Primary Care Team: Wallet Card with Contact Information A customizable wallet-sized card that can be printed, populated with a patient’s primary care team information and given to the patient. The back of the card includes a list of information patients should provide when leaving a message at the clinic. Save a Trip to Primary Care A one-page resource for patients, explaining and illustrating the difference between routine, nonurgent, and urgent medical situations. The sheet explains to patients what actions to take in the case of each of these situations, and provides relevant contact information. View Large Table 1 Care coordination toolkit tool list with descriptions Tool category and tool name Tool description Managing Referrals to Specialty Care VA Service Agreement Template A template with the purpose of facilitating timely access and patient-centered care for patients by promoting an effective relationship between the PACT team and the specialty care team during the management of a patient’s care. Consult Guides for Primary Care (Information to Include) A comprehensive list of clinical conditions and for each, a standard set of information to be included in a referral request, to help ensure a meaningful visit when the referred patient visits the relevant specialist. While the tool consists of multiple data sets, not all need to be implemented. Relaxation and Meditation Program: An approach to self-management of mental health in primary care Information and tools for implementing a program that teaches patients relaxation and meditation skills, with the potential benefit of decreasing the number of referrals to specialty mental health services. Medication Management VA Formulary (Abbreviated) The Primary Care Service in the VA Palo Alto Healthcare System worked with its pharmacy department to create an abbreviated and searchable VA formulary. This resource was posted on the healthcare system’s external web page and is updated periodically for accuracy. AudioRENEWAL: Phone-based Medication Renewal for Patients Allows patients to request a renewal of their prescription directly from within the AudioCARE telephone refill system 24 hr a day, 7 days a week. Communication with Community Providers to Co-Manage Veterans’ Care A two-page document that includes a letter to help facilitate communication between a VA provider and an outside healthcare provider regarding management of the Veteran patient and instructions for obtaining medications through the VA, and a guide for providers (nonVA) to request a prescription medication that is nonformulary. Medication Tracker for Patients A one-page form with sections to be filled in by a member of the PACT team for the patient. Spaces are provided to fill in a patient’s medication details and healthcare provider name and contact information. The medication list includes details that can help patients remember when to take their medications and what dose to take. Patient After-Visit Summary After-Visit Instructions for patients (paper) A customizable document instructing patients about where to go before leaving the VA the day of their appointment, as well as any follow-up actions which need to be taken. Includes contact information for specialty clinics as well as a map of the campus, which can be used to direct patients to their next destination on campus. After-Visit Summary for patients (electronic) An electronic tool that produces a customizable, printable patient summary that can be provided to a patient after his/her primary care office visit to summarize visit content and subsequent action steps, if appropriate. Patient Activation Materials Patient Agenda-Setting Form A simple, easy to fill-out form to help patients prepare for their healthcare visit. The form can be sent to patients in the mail with a pre-visit packet, or can be given to patients to complete in the waiting room. Patient Treatment Decision Guide A worksheet for patients to use when faced with a medical decision. The form includes questions to ask the provider during the medical appointment, space to take notes, and guidance on how to proceed with making a decision. Tips for Patients: Improving Communication with your Primary Care Team A two-page handout that can be mailed to patients in a pre-visit packet or given to patients at check-in, before their appointment. The tool includes tips for communicating with the primary care team, questions for patients to think about before their appointment and space to write down answers. This tool can help ensure all of a patient’s questions and concerns are addressed during their medical appointment. Pharmacy Safety for Patients A patient handout that explains the role of the pharmacist, pharmacy and patient during the process of obtaining medications. Tips for Patients: Questions to Ask Before, During and After your Visit A guide that includes tips, ideas, and questions for patients to use before, during, and after their appointment that will help facilitate a successful healthcare visit. Provider Contact Information for Patients Clinic Information Pamphlet A template that clinics can customize to create a clinic information pamphlet for patients that includes pertinent information about the clinic. Information may include appointment-making instructions, provider contact information, prescription refill instructions, and other clinic details. My Primary Care Team: Contact and Other Information A one-page handout that lists important provider contact information for a patient. Includes contact details for the primary care provider, PACT nurse, tele-health nurse, specialty care, pharmacy, and mental health. My Primary Care Team: Wallet Card with Contact Information A customizable wallet-sized card that can be printed, populated with a patient’s primary care team information and given to the patient. The back of the card includes a list of information patients should provide when leaving a message at the clinic. Save a Trip to Primary Care A one-page resource for patients, explaining and illustrating the difference between routine, nonurgent, and urgent medical situations. The sheet explains to patients what actions to take in the case of each of these situations, and provides relevant contact information. Tool category and tool name Tool description Managing Referrals to Specialty Care VA Service Agreement Template A template with the purpose of facilitating timely access and patient-centered care for patients by promoting an effective relationship between the PACT team and the specialty care team during the management of a patient’s care. Consult Guides for Primary Care (Information to Include) A comprehensive list of clinical conditions and for each, a standard set of information to be included in a referral request, to help ensure a meaningful visit when the referred patient visits the relevant specialist. While the tool consists of multiple data sets, not all need to be implemented. Relaxation and Meditation Program: An approach to self-management of mental health in primary care Information and tools for implementing a program that teaches patients relaxation and meditation skills, with the potential benefit of decreasing the number of referrals to specialty mental health services. Medication Management VA Formulary (Abbreviated) The Primary Care Service in the VA Palo Alto Healthcare System worked with its pharmacy department to create an abbreviated and searchable VA formulary. This resource was posted on the healthcare system’s external web page and is updated periodically for accuracy. AudioRENEWAL: Phone-based Medication Renewal for Patients Allows patients to request a renewal of their prescription directly from within the AudioCARE telephone refill system 24 hr a day, 7 days a week. Communication with Community Providers to Co-Manage Veterans’ Care A two-page document that includes a letter to help facilitate communication between a VA provider and an outside healthcare provider regarding management of the Veteran patient and instructions for obtaining medications through the VA, and a guide for providers (nonVA) to request a prescription medication that is nonformulary. Medication Tracker for Patients A one-page form with sections to be filled in by a member of the PACT team for the patient. Spaces are provided to fill in a patient’s medication details and healthcare provider name and contact information. The medication list includes details that can help patients remember when to take their medications and what dose to take. Patient After-Visit Summary After-Visit Instructions for patients (paper) A customizable document instructing patients about where to go before leaving the VA the day of their appointment, as well as any follow-up actions which need to be taken. Includes contact information for specialty clinics as well as a map of the campus, which can be used to direct patients to their next destination on campus. After-Visit Summary for patients (electronic) An electronic tool that produces a customizable, printable patient summary that can be provided to a patient after his/her primary care office visit to summarize visit content and subsequent action steps, if appropriate. Patient Activation Materials Patient Agenda-Setting Form A simple, easy to fill-out form to help patients prepare for their healthcare visit. The form can be sent to patients in the mail with a pre-visit packet, or can be given to patients to complete in the waiting room. Patient Treatment Decision Guide A worksheet for patients to use when faced with a medical decision. The form includes questions to ask the provider during the medical appointment, space to take notes, and guidance on how to proceed with making a decision. Tips for Patients: Improving Communication with your Primary Care Team A two-page handout that can be mailed to patients in a pre-visit packet or given to patients at check-in, before their appointment. The tool includes tips for communicating with the primary care team, questions for patients to think about before their appointment and space to write down answers. This tool can help ensure all of a patient’s questions and concerns are addressed during their medical appointment. Pharmacy Safety for Patients A patient handout that explains the role of the pharmacist, pharmacy and patient during the process of obtaining medications. Tips for Patients: Questions to Ask Before, During and After your Visit A guide that includes tips, ideas, and questions for patients to use before, during, and after their appointment that will help facilitate a successful healthcare visit. Provider Contact Information for Patients Clinic Information Pamphlet A template that clinics can customize to create a clinic information pamphlet for patients that includes pertinent information about the clinic. Information may include appointment-making instructions, provider contact information, prescription refill instructions, and other clinic details. My Primary Care Team: Contact and Other Information A one-page handout that lists important provider contact information for a patient. Includes contact details for the primary care provider, PACT nurse, tele-health nurse, specialty care, pharmacy, and mental health. My Primary Care Team: Wallet Card with Contact Information A customizable wallet-sized card that can be printed, populated with a patient’s primary care team information and given to the patient. The back of the card includes a list of information patients should provide when leaving a message at the clinic. Save a Trip to Primary Care A one-page resource for patients, explaining and illustrating the difference between routine, nonurgent, and urgent medical situations. The sheet explains to patients what actions to take in the case of each of these situations, and provides relevant contact information. View Large The final set of 18 tools available in the toolkit (available on the VA Intranet, see Supplementary Appendix for screenshots) covers five categories: managing referrals to specialty care, medication management, patient after-visit summary, patient activation materials, and provider contact information for patients. The tools in the “managing referrals to specialty care” category support the appropriate and efficient use of specialty care and better communication between primary care and specialty care teams. The “medication management” category tools can help improve patients’ understanding of how to take their medications or help patients obtain access to needed medications more efficiently. The “patient after-visit summary” category includes tools that offer a convenient way to document for patients the decisions made during their primary care visit and what follow-up is needed. Tools in the category of “patient activation materials” help prepare patients for their visit with their primary care team, and are intended to be sent to (or shared with) the patient in advance of a visit. Finally, the “provider contact information for patients” category tools help to provide patients with important contact information, so they know whom to call for both routine and urgent health issues. The 18 tools vary in the level of resources (e.g. staff time, information technology support) required to implement them. Simpler tools can be used “as is” or with minimal adaptation, and may be implemented and sustained with existing clinic resources. Tools that may be more difficult to implement may require substantial adaptation of the tool for use in the clinic, or the involvement or procurement of new resources such as additional staff, new equipment, or coordination with information technology specialists. An example of an easily implemented tool is “My Primary Care Team – Wallet Card with Contact Information,” a wallet-sized card the clinic prepares in advance and has available for patients when they visit the clinic. The wallet card provides provider names and contact information along with the procedure patients should use when leaving a message at the clinic. An example of a challenging tool to implement is the VA Care Coordination Service Agreement Template, a document that can be used to codify how a primary care team and a specialty care team will coordinate care of shared patients, and which would require substantial negotiation and ongoing meetings between these two teams to be filled out meaningfully. Mapping the tools to available evidence-based care coordination strategies [5] showed good, although not uniform, coverage (Table 2), with clear emphasis on providing support for patients (13 tools), followed by structured arrangements for coordinating service provision between providers (2 tools), providing support for service providers (2 tools), and structuring the relationships between service providers and with patients (1 tool). When accounting for the additional resources available on the toolkit website, coverage was improved, with the strategy of using systems to support care coordination also included. One strategy (improving communication between service providers) was not covered by the tools or additional resources. Table 2 Tools and resources in toolkit, mapped to care coordination strategies in Powell Davies et al. [5] Tool category and tool name Care coordination strategy Managing Referrals to Specialty Care  VA care coordination service agreement template Structured arrangements for coordinating service provision between providers  Consult guides for primary care (information to include) Structured arrangements for coordinating service provision between providers  Relaxation and meditation program: an approach to self-management of mental health in primary care Structuring the relationships between service providers and with patients Medication Management  VA formulary (abbreviated) Providing support for service providers  AudioRENEWAL: Phone-based medication renewal for patients Providing support for patients  Communication with community providers to co-manage Veterans’ care Providing support for service providers  Medication tracker for patients Providing support for patients Patient After-Visit Summary  After-visit instructions for patients (paper) Providing support for patients  After-visit summary for patients (electronic) Providing support for patients Patient Activation Materials  Patient agenda-setting form Providing support for patients  Patient treatment decision guide Providing support for patients  Tips for patients: improving communication with your primary care team Providing support for patients  Pharmacy safety Providing support for patients  Tips for patients: Questions to ask before, during and after your visit Providing support for patients Provider Contact Information for Patients  Clinic information pamphlet Providing support for patients  My primary care team—contact and other information Providing support for patients  My primary care team—wallet card with contact information Providing support for patients  Save a trip to primary care Providing support for patients Additional Resources  Enhancing communication with patients* Providing support for service providers  Care management* Using systems to support care coordination Tool category and tool name Care coordination strategy Managing Referrals to Specialty Care  VA care coordination service agreement template Structured arrangements for coordinating service provision between providers  Consult guides for primary care (information to include) Structured arrangements for coordinating service provision between providers  Relaxation and meditation program: an approach to self-management of mental health in primary care Structuring the relationships between service providers and with patients Medication Management  VA formulary (abbreviated) Providing support for service providers  AudioRENEWAL: Phone-based medication renewal for patients Providing support for patients  Communication with community providers to co-manage Veterans’ care Providing support for service providers  Medication tracker for patients Providing support for patients Patient After-Visit Summary  After-visit instructions for patients (paper) Providing support for patients  After-visit summary for patients (electronic) Providing support for patients Patient Activation Materials  Patient agenda-setting form Providing support for patients  Patient treatment decision guide Providing support for patients  Tips for patients: improving communication with your primary care team Providing support for patients  Pharmacy safety Providing support for patients  Tips for patients: Questions to ask before, during and after your visit Providing support for patients Provider Contact Information for Patients  Clinic information pamphlet Providing support for patients  My primary care team—contact and other information Providing support for patients  My primary care team—wallet card with contact information Providing support for patients  Save a trip to primary care Providing support for patients Additional Resources  Enhancing communication with patients* Providing support for service providers  Care management* Using systems to support care coordination *These are categories of additional resources, with multiple resources within each category. View Large Table 2 Tools and resources in toolkit, mapped to care coordination strategies in Powell Davies et al. [5] Tool category and tool name Care coordination strategy Managing Referrals to Specialty Care  VA care coordination service agreement template Structured arrangements for coordinating service provision between providers  Consult guides for primary care (information to include) Structured arrangements for coordinating service provision between providers  Relaxation and meditation program: an approach to self-management of mental health in primary care Structuring the relationships between service providers and with patients Medication Management  VA formulary (abbreviated) Providing support for service providers  AudioRENEWAL: Phone-based medication renewal for patients Providing support for patients  Communication with community providers to co-manage Veterans’ care Providing support for service providers  Medication tracker for patients Providing support for patients Patient After-Visit Summary  After-visit instructions for patients (paper) Providing support for patients  After-visit summary for patients (electronic) Providing support for patients Patient Activation Materials  Patient agenda-setting form Providing support for patients  Patient treatment decision guide Providing support for patients  Tips for patients: improving communication with your primary care team Providing support for patients  Pharmacy safety Providing support for patients  Tips for patients: Questions to ask before, during and after your visit Providing support for patients Provider Contact Information for Patients  Clinic information pamphlet Providing support for patients  My primary care team—contact and other information Providing support for patients  My primary care team—wallet card with contact information Providing support for patients  Save a trip to primary care Providing support for patients Additional Resources  Enhancing communication with patients* Providing support for service providers  Care management* Using systems to support care coordination Tool category and tool name Care coordination strategy Managing Referrals to Specialty Care  VA care coordination service agreement template Structured arrangements for coordinating service provision between providers  Consult guides for primary care (information to include) Structured arrangements for coordinating service provision between providers  Relaxation and meditation program: an approach to self-management of mental health in primary care Structuring the relationships between service providers and with patients Medication Management  VA formulary (abbreviated) Providing support for service providers  AudioRENEWAL: Phone-based medication renewal for patients Providing support for patients  Communication with community providers to co-manage Veterans’ care Providing support for service providers  Medication tracker for patients Providing support for patients Patient After-Visit Summary  After-visit instructions for patients (paper) Providing support for patients  After-visit summary for patients (electronic) Providing support for patients Patient Activation Materials  Patient agenda-setting form Providing support for patients  Patient treatment decision guide Providing support for patients  Tips for patients: improving communication with your primary care team Providing support for patients  Pharmacy safety Providing support for patients  Tips for patients: Questions to ask before, during and after your visit Providing support for patients Provider Contact Information for Patients  Clinic information pamphlet Providing support for patients  My primary care team—contact and other information Providing support for patients  My primary care team—wallet card with contact information Providing support for patients  Save a trip to primary care Providing support for patients Additional Resources  Enhancing communication with patients* Providing support for service providers  Care management* Using systems to support care coordination *These are categories of additional resources, with multiple resources within each category. View Large Table 3 shows dimensions of patient experience that might be favorably affected by use of the tools if the tools were implemented appropriately. The three problems addressed by the most tools were “difficulty getting questions answered or getting medical advice between scheduled appointments” (six tools), “lack of information about your medical conditions” (five tools), and “lack of information about why your medications have been prescribed to you” (five tools). Mapping tools to SAIL domains showed that the primary domains affected were patient satisfaction and specialty care access (Figure 2). Table 3 Tools in the Care Coordination Toolkit and associated item(s) on the Health System Hassles Scale Tool category and tool name Health System Hassles Scale items 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Managing Referrals to Specialty Care  VA Care Coordination Service Agreement Template X X X  Consult Guides for Primary Care (Information to Include) X X X X X X  Relaxation and Meditation Program: An approach to self-management of mental health in primary care X Medication Management  VA Formulary (Abbreviated) X  AudioRENEWAL: Phone-based Medication Renewal for Patients X X  Communication with Community Providers to Co-Manage Veterans’ Care X X  Medication Tracker for Patients X X Patient After-Visit Summary  After-Visit Instructions for Patients (paper) X X  After Visit Summary for Patients (electronic) X X X Patient Activation Materials  Patient Agenda-Setting Form X X X X X X  Patient Treatment Decision Guide X X  Tips for Patients: Improving Communication with your Primary Care Team X X X X X X  Pharmacy Safety for Patients X  Tips for Patients: Questions to ask Before During and After Your Visit X X X X Provider Contact Information for Patients  Clinic Information Pamphlet X  My Primary Care Team - Contact and Other Information X  My Primary Care Team - Wallet Card with Contact Information X  Save a Trip to Primary Care X X Tool category and tool name Health System Hassles Scale items 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Managing Referrals to Specialty Care  VA Care Coordination Service Agreement Template X X X  Consult Guides for Primary Care (Information to Include) X X X X X X  Relaxation and Meditation Program: An approach to self-management of mental health in primary care X Medication Management  VA Formulary (Abbreviated) X  AudioRENEWAL: Phone-based Medication Renewal for Patients X X  Communication with Community Providers to Co-Manage Veterans’ Care X X  Medication Tracker for Patients X X Patient After-Visit Summary  After-Visit Instructions for Patients (paper) X X  After Visit Summary for Patients (electronic) X X X Patient Activation Materials  Patient Agenda-Setting Form X X X X X X  Patient Treatment Decision Guide X X  Tips for Patients: Improving Communication with your Primary Care Team X X X X X X  Pharmacy Safety for Patients X  Tips for Patients: Questions to ask Before During and After Your Visit X X X X Provider Contact Information for Patients  Clinic Information Pamphlet X  My Primary Care Team - Contact and Other Information X  My Primary Care Team - Wallet Card with Contact Information X  Save a Trip to Primary Care X X Health System Hassles Scale, response options, and 16 items. During the past 12 months, how much of a problem, if at all, has each of the following been for you? (Response options: A very big problem for you, a big problem for you, a moderate problem for you, a small problem for you, or not a problem at all for you.) 1 Lack of information about your medical conditions. 2 Lack of information about which treatment options are best for your medical condition. 3 Lack of information about why your medications have been prescribed to you. 4 Problems getting your medications refilled on time. 5 Uncertainty about when or how to take your medications. 6 Side effects from your medications. 7 Lack of information about why you have been referred to a specialist. 8 Having to wait a long time to get an appointment for specialty providers or clinics. 9 Poor communication between different healthcare providers. 10 Disagreements between your providers about your diagnoses or the best treatment for you. 11 Lack of information about why you need lab tests or x-rays. 12 Having to wait too long to find out about the results of lab tests or x-rays. 13 Difficulty getting questions answered or getting medical advice between scheduled appointments. 14 Lack of time to discuss all your problems with your health care provider during scheduled appointments. 15 Having your concerns ignored or overlooked by your healthcare providers. 16 Medical appointments that interfere with your work, family, or hobbies. View Large Table 3 Tools in the Care Coordination Toolkit and associated item(s) on the Health System Hassles Scale Tool category and tool name Health System Hassles Scale items 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Managing Referrals to Specialty Care  VA Care Coordination Service Agreement Template X X X  Consult Guides for Primary Care (Information to Include) X X X X X X  Relaxation and Meditation Program: An approach to self-management of mental health in primary care X Medication Management  VA Formulary (Abbreviated) X  AudioRENEWAL: Phone-based Medication Renewal for Patients X X  Communication with Community Providers to Co-Manage Veterans’ Care X X  Medication Tracker for Patients X X Patient After-Visit Summary  After-Visit Instructions for Patients (paper) X X  After Visit Summary for Patients (electronic) X X X Patient Activation Materials  Patient Agenda-Setting Form X X X X X X  Patient Treatment Decision Guide X X  Tips for Patients: Improving Communication with your Primary Care Team X X X X X X  Pharmacy Safety for Patients X  Tips for Patients: Questions to ask Before During and After Your Visit X X X X Provider Contact Information for Patients  Clinic Information Pamphlet X  My Primary Care Team - Contact and Other Information X  My Primary Care Team - Wallet Card with Contact Information X  Save a Trip to Primary Care X X Tool category and tool name Health System Hassles Scale items 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Managing Referrals to Specialty Care  VA Care Coordination Service Agreement Template X X X  Consult Guides for Primary Care (Information to Include) X X X X X X  Relaxation and Meditation Program: An approach to self-management of mental health in primary care X Medication Management  VA Formulary (Abbreviated) X  AudioRENEWAL: Phone-based Medication Renewal for Patients X X  Communication with Community Providers to Co-Manage Veterans’ Care X X  Medication Tracker for Patients X X Patient After-Visit Summary  After-Visit Instructions for Patients (paper) X X  After Visit Summary for Patients (electronic) X X X Patient Activation Materials  Patient Agenda-Setting Form X X X X X X  Patient Treatment Decision Guide X X  Tips for Patients: Improving Communication with your Primary Care Team X X X X X X  Pharmacy Safety for Patients X  Tips for Patients: Questions to ask Before During and After Your Visit X X X X Provider Contact Information for Patients  Clinic Information Pamphlet X  My Primary Care Team - Contact and Other Information X  My Primary Care Team - Wallet Card with Contact Information X  Save a Trip to Primary Care X X Health System Hassles Scale, response options, and 16 items. During the past 12 months, how much of a problem, if at all, has each of the following been for you? (Response options: A very big problem for you, a big problem for you, a moderate problem for you, a small problem for you, or not a problem at all for you.) 1 Lack of information about your medical conditions. 2 Lack of information about which treatment options are best for your medical condition. 3 Lack of information about why your medications have been prescribed to you. 4 Problems getting your medications refilled on time. 5 Uncertainty about when or how to take your medications. 6 Side effects from your medications. 7 Lack of information about why you have been referred to a specialist. 8 Having to wait a long time to get an appointment for specialty providers or clinics. 9 Poor communication between different healthcare providers. 10 Disagreements between your providers about your diagnoses or the best treatment for you. 11 Lack of information about why you need lab tests or x-rays. 12 Having to wait too long to find out about the results of lab tests or x-rays. 13 Difficulty getting questions answered or getting medical advice between scheduled appointments. 14 Lack of time to discuss all your problems with your health care provider during scheduled appointments. 15 Having your concerns ignored or overlooked by your healthcare providers. 16 Medical appointments that interfere with your work, family, or hobbies. View Large Fig 2 View largeDownload slide Pathways from tool use to improved dimensions of care quality in the VA. This diagram depicts interactions between the patient and the health system, including primary care (i.e. the “teamlet” which includes one registered nurse care manager, one licensed practical nurse, and one administrative clerk, and one primary care provider; and “the extended PACT team” which can include the addition of a social worker, psychologist, and/or pharmacist). Orange arrows represent tools supporting interactions between parties; bidirectional arrows represent tools requiring input from both parties, whereas uni-directional arrows represent tools where the one party is primarily responsible. Each specific arrow is accompanied by a numbered label that describes the tool and, in italics, the tool’s expected contribution to the overall goal to “facilitate productive interactions.” There are additional yellow markers denoting the SAIL domains that are expected to be positively affected by the productive interactions supported by the tools. Fig 2 View largeDownload slide Pathways from tool use to improved dimensions of care quality in the VA. This diagram depicts interactions between the patient and the health system, including primary care (i.e. the “teamlet” which includes one registered nurse care manager, one licensed practical nurse, and one administrative clerk, and one primary care provider; and “the extended PACT team” which can include the addition of a social worker, psychologist, and/or pharmacist). Orange arrows represent tools supporting interactions between parties; bidirectional arrows represent tools requiring input from both parties, whereas uni-directional arrows represent tools where the one party is primarily responsible. Each specific arrow is accompanied by a numbered label that describes the tool and, in italics, the tool’s expected contribution to the overall goal to “facilitate productive interactions.” There are additional yellow markers denoting the SAIL domains that are expected to be positively affected by the productive interactions supported by the tools. DISCUSSION We have reported on the development process for an online toolkit to support better care coordination in primary care. We were successful in narrowing down a list of 300 potential tools to 18 that met our inclusion criteria and were ultimately made available through our toolkit website. Our two-clinician review process was helpful in deciding on a penultimate list of tools to be reviewed more carefully in semi-structured interviews. Our methods, particularly the tool review form, may be useful to others who are interested in a structured and systematic approach to toolkit development. Most of the tools that were ultimately included in the toolkit related to the coordination strategy of providing support to patients, reflecting the project’s emphasis on improving patient care experiences. Our process also aimed to include tools that were complementary to existing VA resources, rather than duplicative of them. For example, many tools and policies related to how primary care teams should internally coordinate their care were already disseminated in the VA. These aspects of our tool selection process show how the approach adapts to organizational context; in another context, our structured process could be tailored to different opportunities for improvement. Figure 2, adapted from the Chronic Care Model [13], depicts the pathways by which tools and additional resources are ultimately expected to affect aspects of care quality (patient satisfaction and specialty care access) at the VA. Most tool categories work to connect the patient to the health system directly, through increasing accessibility, empowering the patient, improving adherence/self-management, or increasing understanding. One tool category, “managing referrals to specialty care,” operates within the health system, but indirectly impacts productive interactions by improving communication and reducing inefficiencies that impact patient care. The tool review process we carried out is subject to several limitations. First, we attempted to use a quantitative tool review process to determine which tools to subject to further testing; this review process included explicitly rating certain tool features and then providing a summary rating for each tool. Although tool ratings did provide useful information, we learned that they could not be used by themselves for an ultimate determination of each tool’s value. Rather, tools had to be viewed in the context of other features of the project, including the unique features of the VA setting (including whether the VA already supported similar tools via other means), the scope of the CTAC project (including the ability of the tool to be supported by a distance-based coach), and whether the tool was duplicative thematically of other tools already selected for inclusion. As a result, tools that were ultimately included were not uniformly distributed across the various evidence-based strategies for care coordination. Nonetheless, we feel that sufficient variety exists in the toolkit to meet the needs of potential end users in the context of other toolkits available within the VA. Second, we had limited resources to find tools (causing us to use a snowball search strategy) and to interview prospective users about their appropriateness; a more systematic tool identification strategy and a broader group of user interviews might have resulted in a different set of tools being selected. However, our approach may be in line with typical resource constraints faced by organizational improvement teams. Third, the deliberate attempt to choose tools relevant to the VA context and vet them with VA users means that some tools (e.g. VA formulary tool) may not be appropriate for use outside the VA. However, our structured selection process can be used in other organizations. Additionally, because most tools in the toolkit are patient-facing, we believe that many could be used in nonVA settings as is, or with minor adaptations. The field of toolkit development has been hampered by limited evidence about what features of toolkit development lead to effective strategies for implementing evidence-based innovations; in addition, toolkit developers face inherent tradeoffs between deployment time, cost, quality, and flexibility [22]. York and colleagues propose a taxonomy of three different approaches to toolkit development: a “gold standard” approach where candidate tools undergo careful evaluation to ensure that they improve performance; a “grassroots” approach that involves the toolkit creator collecting already-created tools, which then undergo a small amount of vetting; and a “wiki” approach where tools may be posted online without restriction [23]. These different approaches impose various types of burdens on different stakeholders: the gold standard approach may take a long time to develop and make available to users, and result in limited output, but should result in very user-friendly tools that are likely to enhance care; in contrast, the wiki approach is an inexpensive dissemination method but imposes a greater burden on prospective users to sort through the available options. The grassroots approach, although relying on existing materials, is still labor-intensive, as it requires a thorough review (and sometimes outreach) to collect a comprehensive range of tools. Our approach was similar to the grassroots approach chosen by York and colleagues for their toolkits, with respect to the harvesting of existing tools; however, we differed from the grassroots approach in that we used a more extensive review process to ensure saliency of included tools to our users, reduce redundancy, and ensure a minimum level of quality; in addition, we modified tools to improve their look and feel prior to posting them. As a result, we ended up with a smaller number of tools than has been characteristic of prior VA “grassroots” toolkits [23], which may potentially be compensated for by making it easier for prospective users to find and decide upon their preferred tool. Work by DeWalt and colleagues, who carried out a 2-year toolkit development and testing process for the AHRQ Health Literacy Universal Precautions Toolkit, is perhaps the closest to the gold standard articulated above, with draft tools tested in clinical practice [24] and subsequent practice-based evaluations of individual tools and the toolkit collectively [25–27]. Interestingly, the overall demonstration of the toolkit in clinical practice recommended that technical assistance be provided for tool implementation [25]; CTAC intends to test the value of providing such technical assistance for the care coordination toolkit, in the form of a distance-based coach for half of the project sites, while the other half have access to the toolkit only. Although we did not have the time to test tools in clinical practice during the toolkit development phase, we plan to track which tools are adopted (versus never attempted) by each of the participating sites, and among adopted tools, which tools are successfully implemented and maintained (vs. being abandoned), with follow-up key informant interviews exploring reasons for success in implementation or barriers. In addition, we plan to incorporate users’ feedback to refine the tools in the toolkit as they are used, and add new tools when appropriate. We also intend to evaluate the degree to which the toolkit matches the needs of prospective users, the degree to which users’ readiness to implement a tool affects subsequent success in implementation, and the extent to which the tools affect patient experience of care. We attempted to include tools that have varying implementation difficulty, that have broad applicability, and that have not yet been uniformly adopted in the VA based on our semi-structured interviews. However, it remains to be seen whether the tools will successfully align with clinics’ priorities, and there may be a tradeoff in terms of the difficulty of the tools being implemented and the potential beneficial effect on patient experience of care, with more difficult tools potentially providing more benefit, but also requiring more resources. We realized from the outset that the toolkit can be an adjunct to, but not a replacement for, a serious commitment on the part of clinic leadership and staff to improve care coordination. If successful, the CTAC project will improve care coordination in VA primary care clinics, and will also provide readily applicable methods for spreading improvements throughout the VA. In addition, the project will inform VA policymakers regarding what other implementation strategies, including the use of distance coaching, may influence the use of toolkits. SUPPLEMENTARY MATERIAL Supplementary material is available at Translational Behavioral Medicine online. Compliance with ethical standards Conflict of interest: The authors declare that they have no conflicts of interest. Acknowledgments We thank Michael Ong and Adriana Izquierdo for their help in reviewing the tools as expert clinician reviewers. We also thank Tonya Reznor for programming assistance to deploy the toolkit online, John Øvretveit for helpful comments on a previous version of this manuscript, and individuals who participated in semi-structured interviews for their input. This material is based upon work supported by the Department of Veterans Affairs, Quality Enhancement Research Initiative through a grant to the Care Coordination QUERI Program (QUE 15-276). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government. References 1. Berwick DM , Hackbarth AD . Eliminating waste in US health care . Jama . 2012 ; 307 ( 14 ): 1513 – 1516 . Google Scholar CrossRef Search ADS PubMed 2. Berwick DM , Nolan TW , Whittington J . The triple aim: care, health, and cost . Health Aff (Millwood) . 2008 ; 27 ( 3 ): 759 – 769 . Google Scholar CrossRef Search ADS PubMed 3. Gellad WF . The veterans choice act and dual health system use . J Gen Intern Med . 2016 ; 31 ( 2 ): 153 – 154 . Google Scholar CrossRef Search ADS PubMed 4. Yoon J , Scott JY , Phibbs CS , Wagner TH . Recent trends in veterans affairs chronic condition spending . Popul Health Manag . 2011 ; 14 ( 6 ): 293 – 298 . Google Scholar CrossRef Search ADS PubMed 5. Powell Davies G , Williams AM , Larsen K , Perkins D , Roland M , Harris MF . Coordinating primary health care: an analysis of the outcomes of a systematic review . Med J Aust . 2008 ; 188 ( 8 Suppl ): S65 – S68 . Google Scholar PubMed 6. Lau R , Stevenson F , Ong BN , et al. Achieving change in primary care–causes of the evidence to practice gap: systematic reviews of reviews . Implement Sci . 2016 ; 11 : 40 . Google Scholar CrossRef Search ADS PubMed 7. Powell BJ , Waltz TJ , Chinman MJ , et al. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project . Implement Sci . 2015 ; 10 : 21 . Google Scholar CrossRef Search ADS PubMed 8. Gale RC , Asch SM , Taylor T , et al. The most used and most helpful facilitators for patient-centered medical home implementation . Implement Sci . 2015 ; 10 : 52 . Google Scholar CrossRef Search ADS PubMed 9. McDonald KM , Schultz E , Albin L , et al. Care Coordination Measures Atlas Version 4 . Rockville, MD : Agency for Healthcare Research and Quality ; 2014 . 10. AHRQ Publishing and Communications Guidelines, Section 6: Toolkit Guidance . http://www.ahrq.gov/research/publications/pubcomguide/pcguide6.html Accessibility verified January 4, 2018. 11. Bagalman E. The number of veterans that use VA health care services: a fact sheet . Washington, DC : Congressional Research Service ; 2014 . 12. Chang ET , Wang M , Kirsh S , Rubenstein LV. VA High-Risk Populations in Primary Care . Mineapolis, MN : Accepted for presentation at Academy Health Annual Research Meeting ; 2015 . 13. Rosland AM , Nelson K , Sun H , et al. The patient-centered medical home in the Veterans Health Administration . Am J Manag Care . 2013 ; 19 ( 7 ): e263 – 272 . Google Scholar PubMed 14. Moher D , Liberati A , Tetzlaff J , Altman DG ; PRISMA Group . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Plos Med . 2009 ; 6 ( 7 ): e1000097 . Google Scholar CrossRef Search ADS PubMed 15. Luck J , Bowman C , York L , et al. Multimethod evaluation of the VA’s peer-to-peer Toolkit for patient-centered medical home implementation . J Gen Intern Med . 2014 ; 29 ( Suppl 2 ): S572 – S578 . Google Scholar CrossRef Search ADS PubMed 16. Gale RC , Luck J , York L , Asch S . Peer-to-peer toolkits enhance sharing of best practices across an integrated delivery system . J Patient Cent Res Rev . 2016 ; 3 : 189 . Google Scholar CrossRef Search ADS 17. Ganz DA , Huang C , Saliba D , Shier V , Berlowitz D , Lukas CV , Pelczarski K , Schoelles K , Wallace LC , Neumann P. Preventing Falls in Hospitals: A Toolkit for Improving Quality of Care [online] . In. Rockville, MD : Agency for Healthcare Research and Quality ; 2013 . 18. Engaging Patients in Improving Ambulatory Care: A Compendium of Tools from Maine, Oregon, and Humboldt County, California . https://www.rwjf.org/content/dam/farm/toolkits/toolkits/2013/rwjf404402 Accessibility verified January 4, 2018. 19. Carmichael J , Jassar G , Nguyen PA . Healthcare metrics: Where do pharmacists add value ? Am J Health Syst Pharm . 2016 ; 73 ( 19 ): 1537 – 1547 . Google Scholar CrossRef Search ADS PubMed 20. Parchman ML , Noël PH , Lee S . Primary care attributes, health care system hassles, and chronic illness . Med Care . 2005 ; 43 ( 11 ): 1123 – 1129 . Google Scholar CrossRef Search ADS PubMed 21. Tsan MF , Puglisi T . Health care operations activities that may constitute research: the department of veterans affairs’s perspective . Irb . 2014 ; 36 ( 1 ): 9 – 11 . Google Scholar PubMed 22. Reijers HA. Design and Control of Workflow Processes: Business Process Management for the Service Industry. Chapter 6: Heuristic Workflow Redesign , vol. 2617 . Berlin : Springer ; 2003 . 23. York L , Bruce B , Luck J , et al. Online toolkits for metric-driven quality improvement: the veterans health administration managed grassroots approach . Jt Comm J Qual Patient Saf . 2013 ; 39 ( 12 ): 561 – 569 . Google Scholar CrossRef Search ADS PubMed 24. DeWalt DA , Broucksou KA , Hawk V , et al. Developing and testing the health literacy universal precautions toolkit . Nurs Outlook . 2011 ; 59 ( 2 ): 85 – 94 . Google Scholar CrossRef Search ADS PubMed 25. Mabachi NM , Cifuentes M , Barnard J , et al. Demonstration of the health literacy universal precautions toolkit: lessons for quality improvement . J Ambul Care Manage . 2016 ; 39 ( 3 ): 199 – 208 . Google Scholar CrossRef Search ADS PubMed 26. Weiss BD , Brega AG , LeBlanc WG , et al. Improving the effectiveness of medication review: guidance from the health literacy universal precautions toolkit . J Am Board Fam Med . 2016 ; 29 ( 1 ): 18 – 23 . Google Scholar CrossRef Search ADS PubMed 27. Brega AG , Freedman MA , LeBlanc WG , et al. Using the health literacy universal precautions toolkit to improve the quality of patient materials . J Health Commun . 2015 ; 20 ( Suppl 2 ): 69 – 76 . Google Scholar CrossRef Search ADS PubMed Published by Oxford University Press on behalf of the Society of Behavioral Medicine 2018. This work is written by (a) US Government employee(s) and is in the public domain in the US. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Translational Behavioral Medicine Oxford University Press

Loading next page...
 
/lp/ou_press/development-of-a-web-based-toolkit-to-support-improvement-of-care-4DvTmY8W18
Copyright
Published by Oxford University Press on behalf of the Society of Behavioral Medicine 2018.
ISSN
1869-6716
eISSN
1613-9860
D.O.I.
10.1093/tbm/ibx072
Publisher site
See Article on Publisher Site

Abstract

Abstract Promising practices for the coordination of chronic care exist, but how to select and share these practices to support quality improvement within a healthcare system is uncertain. This study describes an approach for selecting high-quality tools for an online care coordination toolkit to be used in Veterans Health Administration (VA) primary care practices. We evaluated tools in three steps: (1) an initial screening to identify tools relevant to care coordination in VA primary care, (2) a two-clinician expert review process assessing tool characteristics (e.g. frequency of problem addressed, linkage to patients’ experience of care, effect on practice workflow, and sustainability with existing resources) and assigning each tool a summary rating, and (3) semi-structured interviews with VA patients and frontline clinicians and staff. Of 300 potentially relevant tools identified by searching online resources, 65, 38, and 18 remained after steps one, two and three, respectively. The 18 tools cover five topics: managing referrals to specialty care, medication management, patient after-visit summary, patient activation materials, agenda setting, patient pre-visit packet, and provider contact information for patients. The final toolkit provides access to the 18 tools, as well as detailed information about tools’ expected benefits, and resources required for tool implementation. Future care coordination efforts can benefit from systematically reviewing available tools to identify those that are high quality and relevant. Implications Practice: To narrow down the very large number of available care coordination tools to those relevant to local improvement goals and context, healthcare leaders and managers can adapt and apply the tool selection process we outline in this article. Policy: A structured approach to toolkit development can facilitate the spread of care innovations for quality improvement efforts, but toolkits are an adjunct to, not a replacement for, a serious commitment on the part of clinic leadership and staff to improve care coordination. Research: To ensure that toolkits are effective in achieving their desired ends, additional work should characterize the development processes and design features of different toolkits, including the relationship between these processes and features and subsequent tool adoption, implementation, and sustainability. INTRODUCTION Inefficiencies in coordination of chronic care are a significant source of waste in the United States (USD $25–$45 billion in 2011) [1] and a barrier to achieving the Triple Aim of improved patient experience of care, lower per capita costs, and better population health [2]. In the Veterans Health Administration (VA), recent legislation allowing some patients to seek care from community medical providers (i.e. outside of VA owned and operated facilities) has further complicated care coordination in the primary care practices of a system where patients already have higher rates of comorbidity and mental illness than private sector counterparts [3, 4]. Although a systematic review has shown the benefits of care coordination strategies in the published research literature [5], effective methods to identify, document, spread, and sustain evidence-based care coordination approaches in routine care are not clear. The Coordination Toolkit and Coaching (CTAC) project aims to improve patients’ experience of care coordination within VA primary care and between VA primary care and other outpatient settings, including health care provided in the community, while also developing better methods for bringing research evidence on care coordination into routine care. To implement research on care coordination in practice, any coordination strategy must not only be evidence-based, but also applicable to the context of the adopting organization. The gap between research and practice exists in part because research results for complex problems such as care coordination are often presented generically, without the supporting documentation of the tools required to achieve such results. Additionally, research results often do not apply when conducted under the varied resource constraints, patient populations, and organizational milieus that are found in practice [6]. These constraints may strongly shape the specific intervention elements that can be applied successfully within any specific care coordination improvement effort. For example, though the VA’s shared information technology infrastructure greatly facilitates internal sharing of innovations, it also means that innovations in research studies or available in general repositories such as the Agency for Healthcare Research and Quality (AHRQ) Innovations Exchange may not be applicable to a local VA initiative because of incompatibility with VA’s electronic health record. In addition, individuals delivering clinical care may have limited time to search for tools, make a comparison between available options, and decide which tools to adopt. Thus, one way to facilitate the uptake of research in practice is to compile a toolkit—in essence, a “short list” of action-oriented strategies for improving care coordination that is compatible with the local context [7]. In this article, we describe CTAC’s first phase, which involved selecting tools for a toolkit and developing a VA Intranet site to support the tools. We aimed to evaluate candidate tools based on the existing evidence base for effective care coordination strategies, relevance to improving patients’ experience of care coordination, tool feasibility and quality, and evaluations by VA patients and frontline VA primary care providers and staff. Within the VA, toolkits are often used in conjunction with other dissemination methods, such as learning collaboratives [8] and quality improvement coaching, and a later phase of CTAC will compare the additional value of distance-based coaching to implement the tools, with access to the toolkit without coaching. METHODS Definitions Care coordination is defined as “…the deliberate organization of patient care activities between two or more participants (including the patient) involved in a patient’s care to facilitate the appropriate delivery of health care services. Organizing care involves the marshalling of personnel and other resources needed to carry out all required patient care activities, and is often managed by the exchange of information among participants responsible for different aspects of care” [9]. A toolkit is defined as “a collection of related information, resources, or tools that together can guide users to develop a plan or organize efforts to follow evidence-based recommendations or meet evidence-based specific practice standards,” and a tool as “an instrument (e.g. survey, guidelines, or checklist) that helps users accomplish a specific task that contributes to meeting a specific evidence-based recommendation or practice standard” [10]. Organizational context The VA is a national integrated healthcare delivery system serving 5.9 million Veteran patients annually [11], with a central headquarters in Washington, DC and 18 healthcare networks known as Veterans Integrated Service Networks (VISNs); each VISN includes multiple healthcare “facilities” that are systems in themselves, offering inpatient and outpatient care, often across multiple sites. In the VA, most patients are assigned to a primary care team [12]. Since 2010, the VA has focused on transforming its primary care teams into a patient-centered medical home model with augmented staffing (one registered nurse care manager, one licensed practical nurse, and one administrative clerk for each primary care provider), called patient-aligned care teams (PACT), and PACTs are tasked with coordinating the patient’s care [13]. VA’s status as an integrated healthcare delivery system offers the benefits of a shared electronic health record and communication systems, including electronic mail, for VA-owned and operated facilities and selected contractors. Toolkit development The decision of which tools to include in the toolkit involved three steps: (1) an initial review of content to determine if tools met inclusion or exclusion criteria, (2) a two-clinician review process to rate tools on their characteristics, and (3) semi-structured interviews with VA patients and frontline clinicians and staff. Figure 1, adapted from the PRISMA statement [14], shows the flow of tools through this three-step process. Fig 1 View largeDownload slide Three-step tool identification, screening and eligibility determination process. 1Preliminary exclusion criteria included a tool being: (1) condition-specific, (2) covering only clinical preventive services, (3) intended for use outside of primary care, (4) not focused on care coordination, (5) not truly a tool, (6) VA already using the tool or strategy widely, (7) not applicable to the VA setting, (8) part of a toolkit and could not stand alone as an individual tool, or (9) not coachable by CTAC’s distance-based coach. 2Exclusions based on the tool-rating checklist (Supplementary Appendix) and reviewer discussion and consensus. 3Excluded tools classified as supporting materials or additional resources for the included tools. Fig 1 View largeDownload slide Three-step tool identification, screening and eligibility determination process. 1Preliminary exclusion criteria included a tool being: (1) condition-specific, (2) covering only clinical preventive services, (3) intended for use outside of primary care, (4) not focused on care coordination, (5) not truly a tool, (6) VA already using the tool or strategy widely, (7) not applicable to the VA setting, (8) part of a toolkit and could not stand alone as an individual tool, or (9) not coachable by CTAC’s distance-based coach. 2Exclusions based on the tool-rating checklist (Supplementary Appendix) and reviewer discussion and consensus. 3Excluded tools classified as supporting materials or additional resources for the included tools. In the first step, the principal investigator and project manager identified tools using a snowball approach, starting with two VA toolkits (focused on PACT and specialty care) [15, 16] and two nonVA toolkits [17, 18]. The focus of this initial review was to determine the candidate tools’ general relevance to the project topic. We excluded tools for any of the following reasons: tools were restricted to a particular clinical domain, such as being condition-specific (e.g. only for diabetes) or covering only clinical preventive services; were intended for use primarily in a setting outside of primary care (e.g. specialty care, inpatient care); were not focused on care coordination (e.g. only access focused, only continuity focused, or only general quality improvement); were not truly a tool (e.g. general information sheets that were not action-oriented); the VA was already disseminating the tool or strategy widely (e.g. tools to encourage uptake of secure messaging between patients and providers); the tool was not applicable to the VA setting; the tool was part of a toolkit with interdependent tools that could not stand alone as individual tools; or CTAC’s distance-based coach would not be able to help primary care teams with using the tool. The principal investigator, project manager, and two additional project staff then sorted remaining tools meeting inclusion criteria into categories and sub-categories to organize them according to the topics they covered. In the second step, two clinicians independently reviewed remaining tools, using a newly developed and piloted tool-rating checklist (Supplementary Appendix). During the second step, the principal investigator served as the first reviewer of all the tools, with the role of second reviewer split between five clinicians, who were assigned roughly equal numbers of tools to review. After the reviews were completed, the first and second reviewers met to discuss substantial differences in ratings (defined as a two-point difference on any 3-point rating of a given dimension of the tool, or at least a 3-point difference on the 7-point tool summary rating). After reviewers discussed these differences, they had the option of changing their ratings if desired. In the third step, the principal investigator (interviewer) and project manager (note-taker) conducted semi-structured interviews with VA providers, clinic staff, and patients in order to obtain feedback about the tools and learn more about how the tools might function in a clinic setting. A total of eight interviews were completed on a broad selection of the included tools, five of them individual interviews with PACT clinic staff from one VA site (primary care provider, registered nurse, lead clinic clerk, psychologist and pharmacist) and three of them group interviews from another site (two Veterans receiving health care services at the VA, members of the 21-member VISN 22 clinician Community of Practice, and six members of the nine-member VISN 22 Nursing Workgroup). Interviews lasted 1 hr each, and were all conducted by phone except for the in-person group interview of the two Veteran patients. A pre-determined selection of tools, specifically chosen for each interviewee based on his/her role at the VA, was then presented, via email attachment, an online screen-sharing program, or in person, as applicable. Interviewees provided feedback about the tools with interviewer prompting via questions such as “Do you have a similar tool in use at your facility?”, “Would a tool like this be helpful at your facility?”, and “How would this tool be used at your facility?” The group interviews of VA providers and staff involved a specific clinical scenario and set of tools that were presented to the group, with the group then providing feedback. After these three steps were completed, final assignment of tools to categories occurred, and tools were deployed to an online website accessible via the VA Intranet, using SharePoint®. Using rapid cycles of development, a SharePoint programmer worked with the principal investigator and project manager to develop the toolkit website’s functionality, usability, and look and feel. An internal SharePoint site was chosen because of rapidity of deployment and the ease with which tools could be added or changed in the future. We placed special emphasis on developing “tool access pages” (Supplementary Appendix), which provide prospective users with a preview of the amount of time and effort required to deploy the tool, and the expected benefits of the tool. Evaluation of toolkit development process To evaluate the process of tool selection and toolkit development, we conducted a review of documentation collected during the process, including spreadsheets of included and excluded tools, reviewer ratings of tools, and field notes from semi-structured interviews. We carried out quantitative analyses of reconciled clinician reviewer scores from the second step of the tool selection process, calculating median and interquartile ranges for tool summary ratings, which were on a 1–7 integer scale (a higher number being more favorable). For inferential statistics, we used the Wilcoxon signed-rank test to assess reviewer agreement on tool ratings, and the Wilcoxon rank-sum test for ratings of included versus excluded tools. Two-tailed p values were considered significant at p < .05. We also evaluated the degree to which the final set of tools selected for inclusion conformed to evidence-based strategies, and the dimensions of health care quality that the tools might affect. To determine the degree to which tools covered the range of evidence-based strategies for care coordination, we mapped the tools to a compilation of care coordination strategies shown effective in a systematic review [5]. We also mapped the pathways by which the selected tools might affect dimensions of the VA’s Strategic Analytics for Improvement and Learning (SAIL) domains, which are an internal dashboard used by VA leaders to benchmark VA facilities’ performance [19]. As the CTAC project focuses on improving patients’ experience of care coordination, we determined in more detail the mapping of tools to the Health System Hassles Scale [20], which assesses patients’ challenges using the healthcare system, such as getting medications refilled on time, lack of information about why patients were referred to a specialist, poor communication between different doctors or clinics, and disagreements between doctors about the patient’s diagnosis or treatment, and is included in the AHRQ Care Coordination Measures Atlas [9]. Ethics approval The CTAC project was determined to be nonresearch by the VA Office of Primary Care Services (in accordance with VHA Handbook 1058.05) [21] and by the VA Greater Los Angeles Healthcare System Institutional Review Board. RESULTS Review of 22 toolkits, manuals and resources (the original four toolkits plus 18 found through a snowball search) resulted in identification of 300 potential tools from these sources (Figure 1). Application of initial inclusion and exclusion criteria narrowed the list of potential tools to 65. The two-clinician review process and subsequent project team discussion further reduced the number of candidate tools to 38. These 38 tools had higher clinician ratings than excluded tools; however, the difference in summary ratings between included and excluded tools was only statistically significantly different for the first reviewer, who reviewed all the tools himself (reviewer 1: median ± interquartile range, 5 ± 1 for included tools vs. 4 ± 1 for excluded tools, p = .0005). For the five different clinicians who shared the role of second reviewer, the collective rating of included and excluded tools showed more variation around median values (reviewer 2: median ± interquartile range, 5 ± 3 for included tools versus 4 ± 2 for excluded tools, p = .096). There was no significant difference in summary ratings between the two reviewers of each tool (p = .915). Semi-structured interviews highlighted factors that might suggest a tool should be included or excluded. For example, one set of tools presented during an interview was focused on motivational interviewing techniques and supporting curricula and resources. The discussion during the interview revealed that training in this area and supplemental materials are already available to VA clinic staff. Other interviews revealed specific challenges a facility might face using a tool. For instance, when presented with an agenda-setting tool that patients receive at check-in and fill out in the waiting room, one interviewee described check-in kiosks used at the clinic, and explained that Veterans do not typically see a clerk if they check in via the kiosk. Therefore, having a clerk hand a paper-based agenda-setting form to the patient at check-in would not be a viable approach in this clinic setting. Based on the feedback from the semi-structured interviews, we were able to reduce the 38 tools at this stage to the final set of 18 (originally 19, with two tools combined into one; Table 1), with many of the excluded 19 tools available on the toolkit website as additional resources or supporting materials for the included tools. Table 1 Care coordination toolkit tool list with descriptions Tool category and tool name Tool description Managing Referrals to Specialty Care VA Service Agreement Template A template with the purpose of facilitating timely access and patient-centered care for patients by promoting an effective relationship between the PACT team and the specialty care team during the management of a patient’s care. Consult Guides for Primary Care (Information to Include) A comprehensive list of clinical conditions and for each, a standard set of information to be included in a referral request, to help ensure a meaningful visit when the referred patient visits the relevant specialist. While the tool consists of multiple data sets, not all need to be implemented. Relaxation and Meditation Program: An approach to self-management of mental health in primary care Information and tools for implementing a program that teaches patients relaxation and meditation skills, with the potential benefit of decreasing the number of referrals to specialty mental health services. Medication Management VA Formulary (Abbreviated) The Primary Care Service in the VA Palo Alto Healthcare System worked with its pharmacy department to create an abbreviated and searchable VA formulary. This resource was posted on the healthcare system’s external web page and is updated periodically for accuracy. AudioRENEWAL: Phone-based Medication Renewal for Patients Allows patients to request a renewal of their prescription directly from within the AudioCARE telephone refill system 24 hr a day, 7 days a week. Communication with Community Providers to Co-Manage Veterans’ Care A two-page document that includes a letter to help facilitate communication between a VA provider and an outside healthcare provider regarding management of the Veteran patient and instructions for obtaining medications through the VA, and a guide for providers (nonVA) to request a prescription medication that is nonformulary. Medication Tracker for Patients A one-page form with sections to be filled in by a member of the PACT team for the patient. Spaces are provided to fill in a patient’s medication details and healthcare provider name and contact information. The medication list includes details that can help patients remember when to take their medications and what dose to take. Patient After-Visit Summary After-Visit Instructions for patients (paper) A customizable document instructing patients about where to go before leaving the VA the day of their appointment, as well as any follow-up actions which need to be taken. Includes contact information for specialty clinics as well as a map of the campus, which can be used to direct patients to their next destination on campus. After-Visit Summary for patients (electronic) An electronic tool that produces a customizable, printable patient summary that can be provided to a patient after his/her primary care office visit to summarize visit content and subsequent action steps, if appropriate. Patient Activation Materials Patient Agenda-Setting Form A simple, easy to fill-out form to help patients prepare for their healthcare visit. The form can be sent to patients in the mail with a pre-visit packet, or can be given to patients to complete in the waiting room. Patient Treatment Decision Guide A worksheet for patients to use when faced with a medical decision. The form includes questions to ask the provider during the medical appointment, space to take notes, and guidance on how to proceed with making a decision. Tips for Patients: Improving Communication with your Primary Care Team A two-page handout that can be mailed to patients in a pre-visit packet or given to patients at check-in, before their appointment. The tool includes tips for communicating with the primary care team, questions for patients to think about before their appointment and space to write down answers. This tool can help ensure all of a patient’s questions and concerns are addressed during their medical appointment. Pharmacy Safety for Patients A patient handout that explains the role of the pharmacist, pharmacy and patient during the process of obtaining medications. Tips for Patients: Questions to Ask Before, During and After your Visit A guide that includes tips, ideas, and questions for patients to use before, during, and after their appointment that will help facilitate a successful healthcare visit. Provider Contact Information for Patients Clinic Information Pamphlet A template that clinics can customize to create a clinic information pamphlet for patients that includes pertinent information about the clinic. Information may include appointment-making instructions, provider contact information, prescription refill instructions, and other clinic details. My Primary Care Team: Contact and Other Information A one-page handout that lists important provider contact information for a patient. Includes contact details for the primary care provider, PACT nurse, tele-health nurse, specialty care, pharmacy, and mental health. My Primary Care Team: Wallet Card with Contact Information A customizable wallet-sized card that can be printed, populated with a patient’s primary care team information and given to the patient. The back of the card includes a list of information patients should provide when leaving a message at the clinic. Save a Trip to Primary Care A one-page resource for patients, explaining and illustrating the difference between routine, nonurgent, and urgent medical situations. The sheet explains to patients what actions to take in the case of each of these situations, and provides relevant contact information. Tool category and tool name Tool description Managing Referrals to Specialty Care VA Service Agreement Template A template with the purpose of facilitating timely access and patient-centered care for patients by promoting an effective relationship between the PACT team and the specialty care team during the management of a patient’s care. Consult Guides for Primary Care (Information to Include) A comprehensive list of clinical conditions and for each, a standard set of information to be included in a referral request, to help ensure a meaningful visit when the referred patient visits the relevant specialist. While the tool consists of multiple data sets, not all need to be implemented. Relaxation and Meditation Program: An approach to self-management of mental health in primary care Information and tools for implementing a program that teaches patients relaxation and meditation skills, with the potential benefit of decreasing the number of referrals to specialty mental health services. Medication Management VA Formulary (Abbreviated) The Primary Care Service in the VA Palo Alto Healthcare System worked with its pharmacy department to create an abbreviated and searchable VA formulary. This resource was posted on the healthcare system’s external web page and is updated periodically for accuracy. AudioRENEWAL: Phone-based Medication Renewal for Patients Allows patients to request a renewal of their prescription directly from within the AudioCARE telephone refill system 24 hr a day, 7 days a week. Communication with Community Providers to Co-Manage Veterans’ Care A two-page document that includes a letter to help facilitate communication between a VA provider and an outside healthcare provider regarding management of the Veteran patient and instructions for obtaining medications through the VA, and a guide for providers (nonVA) to request a prescription medication that is nonformulary. Medication Tracker for Patients A one-page form with sections to be filled in by a member of the PACT team for the patient. Spaces are provided to fill in a patient’s medication details and healthcare provider name and contact information. The medication list includes details that can help patients remember when to take their medications and what dose to take. Patient After-Visit Summary After-Visit Instructions for patients (paper) A customizable document instructing patients about where to go before leaving the VA the day of their appointment, as well as any follow-up actions which need to be taken. Includes contact information for specialty clinics as well as a map of the campus, which can be used to direct patients to their next destination on campus. After-Visit Summary for patients (electronic) An electronic tool that produces a customizable, printable patient summary that can be provided to a patient after his/her primary care office visit to summarize visit content and subsequent action steps, if appropriate. Patient Activation Materials Patient Agenda-Setting Form A simple, easy to fill-out form to help patients prepare for their healthcare visit. The form can be sent to patients in the mail with a pre-visit packet, or can be given to patients to complete in the waiting room. Patient Treatment Decision Guide A worksheet for patients to use when faced with a medical decision. The form includes questions to ask the provider during the medical appointment, space to take notes, and guidance on how to proceed with making a decision. Tips for Patients: Improving Communication with your Primary Care Team A two-page handout that can be mailed to patients in a pre-visit packet or given to patients at check-in, before their appointment. The tool includes tips for communicating with the primary care team, questions for patients to think about before their appointment and space to write down answers. This tool can help ensure all of a patient’s questions and concerns are addressed during their medical appointment. Pharmacy Safety for Patients A patient handout that explains the role of the pharmacist, pharmacy and patient during the process of obtaining medications. Tips for Patients: Questions to Ask Before, During and After your Visit A guide that includes tips, ideas, and questions for patients to use before, during, and after their appointment that will help facilitate a successful healthcare visit. Provider Contact Information for Patients Clinic Information Pamphlet A template that clinics can customize to create a clinic information pamphlet for patients that includes pertinent information about the clinic. Information may include appointment-making instructions, provider contact information, prescription refill instructions, and other clinic details. My Primary Care Team: Contact and Other Information A one-page handout that lists important provider contact information for a patient. Includes contact details for the primary care provider, PACT nurse, tele-health nurse, specialty care, pharmacy, and mental health. My Primary Care Team: Wallet Card with Contact Information A customizable wallet-sized card that can be printed, populated with a patient’s primary care team information and given to the patient. The back of the card includes a list of information patients should provide when leaving a message at the clinic. Save a Trip to Primary Care A one-page resource for patients, explaining and illustrating the difference between routine, nonurgent, and urgent medical situations. The sheet explains to patients what actions to take in the case of each of these situations, and provides relevant contact information. View Large Table 1 Care coordination toolkit tool list with descriptions Tool category and tool name Tool description Managing Referrals to Specialty Care VA Service Agreement Template A template with the purpose of facilitating timely access and patient-centered care for patients by promoting an effective relationship between the PACT team and the specialty care team during the management of a patient’s care. Consult Guides for Primary Care (Information to Include) A comprehensive list of clinical conditions and for each, a standard set of information to be included in a referral request, to help ensure a meaningful visit when the referred patient visits the relevant specialist. While the tool consists of multiple data sets, not all need to be implemented. Relaxation and Meditation Program: An approach to self-management of mental health in primary care Information and tools for implementing a program that teaches patients relaxation and meditation skills, with the potential benefit of decreasing the number of referrals to specialty mental health services. Medication Management VA Formulary (Abbreviated) The Primary Care Service in the VA Palo Alto Healthcare System worked with its pharmacy department to create an abbreviated and searchable VA formulary. This resource was posted on the healthcare system’s external web page and is updated periodically for accuracy. AudioRENEWAL: Phone-based Medication Renewal for Patients Allows patients to request a renewal of their prescription directly from within the AudioCARE telephone refill system 24 hr a day, 7 days a week. Communication with Community Providers to Co-Manage Veterans’ Care A two-page document that includes a letter to help facilitate communication between a VA provider and an outside healthcare provider regarding management of the Veteran patient and instructions for obtaining medications through the VA, and a guide for providers (nonVA) to request a prescription medication that is nonformulary. Medication Tracker for Patients A one-page form with sections to be filled in by a member of the PACT team for the patient. Spaces are provided to fill in a patient’s medication details and healthcare provider name and contact information. The medication list includes details that can help patients remember when to take their medications and what dose to take. Patient After-Visit Summary After-Visit Instructions for patients (paper) A customizable document instructing patients about where to go before leaving the VA the day of their appointment, as well as any follow-up actions which need to be taken. Includes contact information for specialty clinics as well as a map of the campus, which can be used to direct patients to their next destination on campus. After-Visit Summary for patients (electronic) An electronic tool that produces a customizable, printable patient summary that can be provided to a patient after his/her primary care office visit to summarize visit content and subsequent action steps, if appropriate. Patient Activation Materials Patient Agenda-Setting Form A simple, easy to fill-out form to help patients prepare for their healthcare visit. The form can be sent to patients in the mail with a pre-visit packet, or can be given to patients to complete in the waiting room. Patient Treatment Decision Guide A worksheet for patients to use when faced with a medical decision. The form includes questions to ask the provider during the medical appointment, space to take notes, and guidance on how to proceed with making a decision. Tips for Patients: Improving Communication with your Primary Care Team A two-page handout that can be mailed to patients in a pre-visit packet or given to patients at check-in, before their appointment. The tool includes tips for communicating with the primary care team, questions for patients to think about before their appointment and space to write down answers. This tool can help ensure all of a patient’s questions and concerns are addressed during their medical appointment. Pharmacy Safety for Patients A patient handout that explains the role of the pharmacist, pharmacy and patient during the process of obtaining medications. Tips for Patients: Questions to Ask Before, During and After your Visit A guide that includes tips, ideas, and questions for patients to use before, during, and after their appointment that will help facilitate a successful healthcare visit. Provider Contact Information for Patients Clinic Information Pamphlet A template that clinics can customize to create a clinic information pamphlet for patients that includes pertinent information about the clinic. Information may include appointment-making instructions, provider contact information, prescription refill instructions, and other clinic details. My Primary Care Team: Contact and Other Information A one-page handout that lists important provider contact information for a patient. Includes contact details for the primary care provider, PACT nurse, tele-health nurse, specialty care, pharmacy, and mental health. My Primary Care Team: Wallet Card with Contact Information A customizable wallet-sized card that can be printed, populated with a patient’s primary care team information and given to the patient. The back of the card includes a list of information patients should provide when leaving a message at the clinic. Save a Trip to Primary Care A one-page resource for patients, explaining and illustrating the difference between routine, nonurgent, and urgent medical situations. The sheet explains to patients what actions to take in the case of each of these situations, and provides relevant contact information. Tool category and tool name Tool description Managing Referrals to Specialty Care VA Service Agreement Template A template with the purpose of facilitating timely access and patient-centered care for patients by promoting an effective relationship between the PACT team and the specialty care team during the management of a patient’s care. Consult Guides for Primary Care (Information to Include) A comprehensive list of clinical conditions and for each, a standard set of information to be included in a referral request, to help ensure a meaningful visit when the referred patient visits the relevant specialist. While the tool consists of multiple data sets, not all need to be implemented. Relaxation and Meditation Program: An approach to self-management of mental health in primary care Information and tools for implementing a program that teaches patients relaxation and meditation skills, with the potential benefit of decreasing the number of referrals to specialty mental health services. Medication Management VA Formulary (Abbreviated) The Primary Care Service in the VA Palo Alto Healthcare System worked with its pharmacy department to create an abbreviated and searchable VA formulary. This resource was posted on the healthcare system’s external web page and is updated periodically for accuracy. AudioRENEWAL: Phone-based Medication Renewal for Patients Allows patients to request a renewal of their prescription directly from within the AudioCARE telephone refill system 24 hr a day, 7 days a week. Communication with Community Providers to Co-Manage Veterans’ Care A two-page document that includes a letter to help facilitate communication between a VA provider and an outside healthcare provider regarding management of the Veteran patient and instructions for obtaining medications through the VA, and a guide for providers (nonVA) to request a prescription medication that is nonformulary. Medication Tracker for Patients A one-page form with sections to be filled in by a member of the PACT team for the patient. Spaces are provided to fill in a patient’s medication details and healthcare provider name and contact information. The medication list includes details that can help patients remember when to take their medications and what dose to take. Patient After-Visit Summary After-Visit Instructions for patients (paper) A customizable document instructing patients about where to go before leaving the VA the day of their appointment, as well as any follow-up actions which need to be taken. Includes contact information for specialty clinics as well as a map of the campus, which can be used to direct patients to their next destination on campus. After-Visit Summary for patients (electronic) An electronic tool that produces a customizable, printable patient summary that can be provided to a patient after his/her primary care office visit to summarize visit content and subsequent action steps, if appropriate. Patient Activation Materials Patient Agenda-Setting Form A simple, easy to fill-out form to help patients prepare for their healthcare visit. The form can be sent to patients in the mail with a pre-visit packet, or can be given to patients to complete in the waiting room. Patient Treatment Decision Guide A worksheet for patients to use when faced with a medical decision. The form includes questions to ask the provider during the medical appointment, space to take notes, and guidance on how to proceed with making a decision. Tips for Patients: Improving Communication with your Primary Care Team A two-page handout that can be mailed to patients in a pre-visit packet or given to patients at check-in, before their appointment. The tool includes tips for communicating with the primary care team, questions for patients to think about before their appointment and space to write down answers. This tool can help ensure all of a patient’s questions and concerns are addressed during their medical appointment. Pharmacy Safety for Patients A patient handout that explains the role of the pharmacist, pharmacy and patient during the process of obtaining medications. Tips for Patients: Questions to Ask Before, During and After your Visit A guide that includes tips, ideas, and questions for patients to use before, during, and after their appointment that will help facilitate a successful healthcare visit. Provider Contact Information for Patients Clinic Information Pamphlet A template that clinics can customize to create a clinic information pamphlet for patients that includes pertinent information about the clinic. Information may include appointment-making instructions, provider contact information, prescription refill instructions, and other clinic details. My Primary Care Team: Contact and Other Information A one-page handout that lists important provider contact information for a patient. Includes contact details for the primary care provider, PACT nurse, tele-health nurse, specialty care, pharmacy, and mental health. My Primary Care Team: Wallet Card with Contact Information A customizable wallet-sized card that can be printed, populated with a patient’s primary care team information and given to the patient. The back of the card includes a list of information patients should provide when leaving a message at the clinic. Save a Trip to Primary Care A one-page resource for patients, explaining and illustrating the difference between routine, nonurgent, and urgent medical situations. The sheet explains to patients what actions to take in the case of each of these situations, and provides relevant contact information. View Large The final set of 18 tools available in the toolkit (available on the VA Intranet, see Supplementary Appendix for screenshots) covers five categories: managing referrals to specialty care, medication management, patient after-visit summary, patient activation materials, and provider contact information for patients. The tools in the “managing referrals to specialty care” category support the appropriate and efficient use of specialty care and better communication between primary care and specialty care teams. The “medication management” category tools can help improve patients’ understanding of how to take their medications or help patients obtain access to needed medications more efficiently. The “patient after-visit summary” category includes tools that offer a convenient way to document for patients the decisions made during their primary care visit and what follow-up is needed. Tools in the category of “patient activation materials” help prepare patients for their visit with their primary care team, and are intended to be sent to (or shared with) the patient in advance of a visit. Finally, the “provider contact information for patients” category tools help to provide patients with important contact information, so they know whom to call for both routine and urgent health issues. The 18 tools vary in the level of resources (e.g. staff time, information technology support) required to implement them. Simpler tools can be used “as is” or with minimal adaptation, and may be implemented and sustained with existing clinic resources. Tools that may be more difficult to implement may require substantial adaptation of the tool for use in the clinic, or the involvement or procurement of new resources such as additional staff, new equipment, or coordination with information technology specialists. An example of an easily implemented tool is “My Primary Care Team – Wallet Card with Contact Information,” a wallet-sized card the clinic prepares in advance and has available for patients when they visit the clinic. The wallet card provides provider names and contact information along with the procedure patients should use when leaving a message at the clinic. An example of a challenging tool to implement is the VA Care Coordination Service Agreement Template, a document that can be used to codify how a primary care team and a specialty care team will coordinate care of shared patients, and which would require substantial negotiation and ongoing meetings between these two teams to be filled out meaningfully. Mapping the tools to available evidence-based care coordination strategies [5] showed good, although not uniform, coverage (Table 2), with clear emphasis on providing support for patients (13 tools), followed by structured arrangements for coordinating service provision between providers (2 tools), providing support for service providers (2 tools), and structuring the relationships between service providers and with patients (1 tool). When accounting for the additional resources available on the toolkit website, coverage was improved, with the strategy of using systems to support care coordination also included. One strategy (improving communication between service providers) was not covered by the tools or additional resources. Table 2 Tools and resources in toolkit, mapped to care coordination strategies in Powell Davies et al. [5] Tool category and tool name Care coordination strategy Managing Referrals to Specialty Care  VA care coordination service agreement template Structured arrangements for coordinating service provision between providers  Consult guides for primary care (information to include) Structured arrangements for coordinating service provision between providers  Relaxation and meditation program: an approach to self-management of mental health in primary care Structuring the relationships between service providers and with patients Medication Management  VA formulary (abbreviated) Providing support for service providers  AudioRENEWAL: Phone-based medication renewal for patients Providing support for patients  Communication with community providers to co-manage Veterans’ care Providing support for service providers  Medication tracker for patients Providing support for patients Patient After-Visit Summary  After-visit instructions for patients (paper) Providing support for patients  After-visit summary for patients (electronic) Providing support for patients Patient Activation Materials  Patient agenda-setting form Providing support for patients  Patient treatment decision guide Providing support for patients  Tips for patients: improving communication with your primary care team Providing support for patients  Pharmacy safety Providing support for patients  Tips for patients: Questions to ask before, during and after your visit Providing support for patients Provider Contact Information for Patients  Clinic information pamphlet Providing support for patients  My primary care team—contact and other information Providing support for patients  My primary care team—wallet card with contact information Providing support for patients  Save a trip to primary care Providing support for patients Additional Resources  Enhancing communication with patients* Providing support for service providers  Care management* Using systems to support care coordination Tool category and tool name Care coordination strategy Managing Referrals to Specialty Care  VA care coordination service agreement template Structured arrangements for coordinating service provision between providers  Consult guides for primary care (information to include) Structured arrangements for coordinating service provision between providers  Relaxation and meditation program: an approach to self-management of mental health in primary care Structuring the relationships between service providers and with patients Medication Management  VA formulary (abbreviated) Providing support for service providers  AudioRENEWAL: Phone-based medication renewal for patients Providing support for patients  Communication with community providers to co-manage Veterans’ care Providing support for service providers  Medication tracker for patients Providing support for patients Patient After-Visit Summary  After-visit instructions for patients (paper) Providing support for patients  After-visit summary for patients (electronic) Providing support for patients Patient Activation Materials  Patient agenda-setting form Providing support for patients  Patient treatment decision guide Providing support for patients  Tips for patients: improving communication with your primary care team Providing support for patients  Pharmacy safety Providing support for patients  Tips for patients: Questions to ask before, during and after your visit Providing support for patients Provider Contact Information for Patients  Clinic information pamphlet Providing support for patients  My primary care team—contact and other information Providing support for patients  My primary care team—wallet card with contact information Providing support for patients  Save a trip to primary care Providing support for patients Additional Resources  Enhancing communication with patients* Providing support for service providers  Care management* Using systems to support care coordination *These are categories of additional resources, with multiple resources within each category. View Large Table 2 Tools and resources in toolkit, mapped to care coordination strategies in Powell Davies et al. [5] Tool category and tool name Care coordination strategy Managing Referrals to Specialty Care  VA care coordination service agreement template Structured arrangements for coordinating service provision between providers  Consult guides for primary care (information to include) Structured arrangements for coordinating service provision between providers  Relaxation and meditation program: an approach to self-management of mental health in primary care Structuring the relationships between service providers and with patients Medication Management  VA formulary (abbreviated) Providing support for service providers  AudioRENEWAL: Phone-based medication renewal for patients Providing support for patients  Communication with community providers to co-manage Veterans’ care Providing support for service providers  Medication tracker for patients Providing support for patients Patient After-Visit Summary  After-visit instructions for patients (paper) Providing support for patients  After-visit summary for patients (electronic) Providing support for patients Patient Activation Materials  Patient agenda-setting form Providing support for patients  Patient treatment decision guide Providing support for patients  Tips for patients: improving communication with your primary care team Providing support for patients  Pharmacy safety Providing support for patients  Tips for patients: Questions to ask before, during and after your visit Providing support for patients Provider Contact Information for Patients  Clinic information pamphlet Providing support for patients  My primary care team—contact and other information Providing support for patients  My primary care team—wallet card with contact information Providing support for patients  Save a trip to primary care Providing support for patients Additional Resources  Enhancing communication with patients* Providing support for service providers  Care management* Using systems to support care coordination Tool category and tool name Care coordination strategy Managing Referrals to Specialty Care  VA care coordination service agreement template Structured arrangements for coordinating service provision between providers  Consult guides for primary care (information to include) Structured arrangements for coordinating service provision between providers  Relaxation and meditation program: an approach to self-management of mental health in primary care Structuring the relationships between service providers and with patients Medication Management  VA formulary (abbreviated) Providing support for service providers  AudioRENEWAL: Phone-based medication renewal for patients Providing support for patients  Communication with community providers to co-manage Veterans’ care Providing support for service providers  Medication tracker for patients Providing support for patients Patient After-Visit Summary  After-visit instructions for patients (paper) Providing support for patients  After-visit summary for patients (electronic) Providing support for patients Patient Activation Materials  Patient agenda-setting form Providing support for patients  Patient treatment decision guide Providing support for patients  Tips for patients: improving communication with your primary care team Providing support for patients  Pharmacy safety Providing support for patients  Tips for patients: Questions to ask before, during and after your visit Providing support for patients Provider Contact Information for Patients  Clinic information pamphlet Providing support for patients  My primary care team—contact and other information Providing support for patients  My primary care team—wallet card with contact information Providing support for patients  Save a trip to primary care Providing support for patients Additional Resources  Enhancing communication with patients* Providing support for service providers  Care management* Using systems to support care coordination *These are categories of additional resources, with multiple resources within each category. View Large Table 3 shows dimensions of patient experience that might be favorably affected by use of the tools if the tools were implemented appropriately. The three problems addressed by the most tools were “difficulty getting questions answered or getting medical advice between scheduled appointments” (six tools), “lack of information about your medical conditions” (five tools), and “lack of information about why your medications have been prescribed to you” (five tools). Mapping tools to SAIL domains showed that the primary domains affected were patient satisfaction and specialty care access (Figure 2). Table 3 Tools in the Care Coordination Toolkit and associated item(s) on the Health System Hassles Scale Tool category and tool name Health System Hassles Scale items 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Managing Referrals to Specialty Care  VA Care Coordination Service Agreement Template X X X  Consult Guides for Primary Care (Information to Include) X X X X X X  Relaxation and Meditation Program: An approach to self-management of mental health in primary care X Medication Management  VA Formulary (Abbreviated) X  AudioRENEWAL: Phone-based Medication Renewal for Patients X X  Communication with Community Providers to Co-Manage Veterans’ Care X X  Medication Tracker for Patients X X Patient After-Visit Summary  After-Visit Instructions for Patients (paper) X X  After Visit Summary for Patients (electronic) X X X Patient Activation Materials  Patient Agenda-Setting Form X X X X X X  Patient Treatment Decision Guide X X  Tips for Patients: Improving Communication with your Primary Care Team X X X X X X  Pharmacy Safety for Patients X  Tips for Patients: Questions to ask Before During and After Your Visit X X X X Provider Contact Information for Patients  Clinic Information Pamphlet X  My Primary Care Team - Contact and Other Information X  My Primary Care Team - Wallet Card with Contact Information X  Save a Trip to Primary Care X X Tool category and tool name Health System Hassles Scale items 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Managing Referrals to Specialty Care  VA Care Coordination Service Agreement Template X X X  Consult Guides for Primary Care (Information to Include) X X X X X X  Relaxation and Meditation Program: An approach to self-management of mental health in primary care X Medication Management  VA Formulary (Abbreviated) X  AudioRENEWAL: Phone-based Medication Renewal for Patients X X  Communication with Community Providers to Co-Manage Veterans’ Care X X  Medication Tracker for Patients X X Patient After-Visit Summary  After-Visit Instructions for Patients (paper) X X  After Visit Summary for Patients (electronic) X X X Patient Activation Materials  Patient Agenda-Setting Form X X X X X X  Patient Treatment Decision Guide X X  Tips for Patients: Improving Communication with your Primary Care Team X X X X X X  Pharmacy Safety for Patients X  Tips for Patients: Questions to ask Before During and After Your Visit X X X X Provider Contact Information for Patients  Clinic Information Pamphlet X  My Primary Care Team - Contact and Other Information X  My Primary Care Team - Wallet Card with Contact Information X  Save a Trip to Primary Care X X Health System Hassles Scale, response options, and 16 items. During the past 12 months, how much of a problem, if at all, has each of the following been for you? (Response options: A very big problem for you, a big problem for you, a moderate problem for you, a small problem for you, or not a problem at all for you.) 1 Lack of information about your medical conditions. 2 Lack of information about which treatment options are best for your medical condition. 3 Lack of information about why your medications have been prescribed to you. 4 Problems getting your medications refilled on time. 5 Uncertainty about when or how to take your medications. 6 Side effects from your medications. 7 Lack of information about why you have been referred to a specialist. 8 Having to wait a long time to get an appointment for specialty providers or clinics. 9 Poor communication between different healthcare providers. 10 Disagreements between your providers about your diagnoses or the best treatment for you. 11 Lack of information about why you need lab tests or x-rays. 12 Having to wait too long to find out about the results of lab tests or x-rays. 13 Difficulty getting questions answered or getting medical advice between scheduled appointments. 14 Lack of time to discuss all your problems with your health care provider during scheduled appointments. 15 Having your concerns ignored or overlooked by your healthcare providers. 16 Medical appointments that interfere with your work, family, or hobbies. View Large Table 3 Tools in the Care Coordination Toolkit and associated item(s) on the Health System Hassles Scale Tool category and tool name Health System Hassles Scale items 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Managing Referrals to Specialty Care  VA Care Coordination Service Agreement Template X X X  Consult Guides for Primary Care (Information to Include) X X X X X X  Relaxation and Meditation Program: An approach to self-management of mental health in primary care X Medication Management  VA Formulary (Abbreviated) X  AudioRENEWAL: Phone-based Medication Renewal for Patients X X  Communication with Community Providers to Co-Manage Veterans’ Care X X  Medication Tracker for Patients X X Patient After-Visit Summary  After-Visit Instructions for Patients (paper) X X  After Visit Summary for Patients (electronic) X X X Patient Activation Materials  Patient Agenda-Setting Form X X X X X X  Patient Treatment Decision Guide X X  Tips for Patients: Improving Communication with your Primary Care Team X X X X X X  Pharmacy Safety for Patients X  Tips for Patients: Questions to ask Before During and After Your Visit X X X X Provider Contact Information for Patients  Clinic Information Pamphlet X  My Primary Care Team - Contact and Other Information X  My Primary Care Team - Wallet Card with Contact Information X  Save a Trip to Primary Care X X Tool category and tool name Health System Hassles Scale items 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Managing Referrals to Specialty Care  VA Care Coordination Service Agreement Template X X X  Consult Guides for Primary Care (Information to Include) X X X X X X  Relaxation and Meditation Program: An approach to self-management of mental health in primary care X Medication Management  VA Formulary (Abbreviated) X  AudioRENEWAL: Phone-based Medication Renewal for Patients X X  Communication with Community Providers to Co-Manage Veterans’ Care X X  Medication Tracker for Patients X X Patient After-Visit Summary  After-Visit Instructions for Patients (paper) X X  After Visit Summary for Patients (electronic) X X X Patient Activation Materials  Patient Agenda-Setting Form X X X X X X  Patient Treatment Decision Guide X X  Tips for Patients: Improving Communication with your Primary Care Team X X X X X X  Pharmacy Safety for Patients X  Tips for Patients: Questions to ask Before During and After Your Visit X X X X Provider Contact Information for Patients  Clinic Information Pamphlet X  My Primary Care Team - Contact and Other Information X  My Primary Care Team - Wallet Card with Contact Information X  Save a Trip to Primary Care X X Health System Hassles Scale, response options, and 16 items. During the past 12 months, how much of a problem, if at all, has each of the following been for you? (Response options: A very big problem for you, a big problem for you, a moderate problem for you, a small problem for you, or not a problem at all for you.) 1 Lack of information about your medical conditions. 2 Lack of information about which treatment options are best for your medical condition. 3 Lack of information about why your medications have been prescribed to you. 4 Problems getting your medications refilled on time. 5 Uncertainty about when or how to take your medications. 6 Side effects from your medications. 7 Lack of information about why you have been referred to a specialist. 8 Having to wait a long time to get an appointment for specialty providers or clinics. 9 Poor communication between different healthcare providers. 10 Disagreements between your providers about your diagnoses or the best treatment for you. 11 Lack of information about why you need lab tests or x-rays. 12 Having to wait too long to find out about the results of lab tests or x-rays. 13 Difficulty getting questions answered or getting medical advice between scheduled appointments. 14 Lack of time to discuss all your problems with your health care provider during scheduled appointments. 15 Having your concerns ignored or overlooked by your healthcare providers. 16 Medical appointments that interfere with your work, family, or hobbies. View Large Fig 2 View largeDownload slide Pathways from tool use to improved dimensions of care quality in the VA. This diagram depicts interactions between the patient and the health system, including primary care (i.e. the “teamlet” which includes one registered nurse care manager, one licensed practical nurse, and one administrative clerk, and one primary care provider; and “the extended PACT team” which can include the addition of a social worker, psychologist, and/or pharmacist). Orange arrows represent tools supporting interactions between parties; bidirectional arrows represent tools requiring input from both parties, whereas uni-directional arrows represent tools where the one party is primarily responsible. Each specific arrow is accompanied by a numbered label that describes the tool and, in italics, the tool’s expected contribution to the overall goal to “facilitate productive interactions.” There are additional yellow markers denoting the SAIL domains that are expected to be positively affected by the productive interactions supported by the tools. Fig 2 View largeDownload slide Pathways from tool use to improved dimensions of care quality in the VA. This diagram depicts interactions between the patient and the health system, including primary care (i.e. the “teamlet” which includes one registered nurse care manager, one licensed practical nurse, and one administrative clerk, and one primary care provider; and “the extended PACT team” which can include the addition of a social worker, psychologist, and/or pharmacist). Orange arrows represent tools supporting interactions between parties; bidirectional arrows represent tools requiring input from both parties, whereas uni-directional arrows represent tools where the one party is primarily responsible. Each specific arrow is accompanied by a numbered label that describes the tool and, in italics, the tool’s expected contribution to the overall goal to “facilitate productive interactions.” There are additional yellow markers denoting the SAIL domains that are expected to be positively affected by the productive interactions supported by the tools. DISCUSSION We have reported on the development process for an online toolkit to support better care coordination in primary care. We were successful in narrowing down a list of 300 potential tools to 18 that met our inclusion criteria and were ultimately made available through our toolkit website. Our two-clinician review process was helpful in deciding on a penultimate list of tools to be reviewed more carefully in semi-structured interviews. Our methods, particularly the tool review form, may be useful to others who are interested in a structured and systematic approach to toolkit development. Most of the tools that were ultimately included in the toolkit related to the coordination strategy of providing support to patients, reflecting the project’s emphasis on improving patient care experiences. Our process also aimed to include tools that were complementary to existing VA resources, rather than duplicative of them. For example, many tools and policies related to how primary care teams should internally coordinate their care were already disseminated in the VA. These aspects of our tool selection process show how the approach adapts to organizational context; in another context, our structured process could be tailored to different opportunities for improvement. Figure 2, adapted from the Chronic Care Model [13], depicts the pathways by which tools and additional resources are ultimately expected to affect aspects of care quality (patient satisfaction and specialty care access) at the VA. Most tool categories work to connect the patient to the health system directly, through increasing accessibility, empowering the patient, improving adherence/self-management, or increasing understanding. One tool category, “managing referrals to specialty care,” operates within the health system, but indirectly impacts productive interactions by improving communication and reducing inefficiencies that impact patient care. The tool review process we carried out is subject to several limitations. First, we attempted to use a quantitative tool review process to determine which tools to subject to further testing; this review process included explicitly rating certain tool features and then providing a summary rating for each tool. Although tool ratings did provide useful information, we learned that they could not be used by themselves for an ultimate determination of each tool’s value. Rather, tools had to be viewed in the context of other features of the project, including the unique features of the VA setting (including whether the VA already supported similar tools via other means), the scope of the CTAC project (including the ability of the tool to be supported by a distance-based coach), and whether the tool was duplicative thematically of other tools already selected for inclusion. As a result, tools that were ultimately included were not uniformly distributed across the various evidence-based strategies for care coordination. Nonetheless, we feel that sufficient variety exists in the toolkit to meet the needs of potential end users in the context of other toolkits available within the VA. Second, we had limited resources to find tools (causing us to use a snowball search strategy) and to interview prospective users about their appropriateness; a more systematic tool identification strategy and a broader group of user interviews might have resulted in a different set of tools being selected. However, our approach may be in line with typical resource constraints faced by organizational improvement teams. Third, the deliberate attempt to choose tools relevant to the VA context and vet them with VA users means that some tools (e.g. VA formulary tool) may not be appropriate for use outside the VA. However, our structured selection process can be used in other organizations. Additionally, because most tools in the toolkit are patient-facing, we believe that many could be used in nonVA settings as is, or with minor adaptations. The field of toolkit development has been hampered by limited evidence about what features of toolkit development lead to effective strategies for implementing evidence-based innovations; in addition, toolkit developers face inherent tradeoffs between deployment time, cost, quality, and flexibility [22]. York and colleagues propose a taxonomy of three different approaches to toolkit development: a “gold standard” approach where candidate tools undergo careful evaluation to ensure that they improve performance; a “grassroots” approach that involves the toolkit creator collecting already-created tools, which then undergo a small amount of vetting; and a “wiki” approach where tools may be posted online without restriction [23]. These different approaches impose various types of burdens on different stakeholders: the gold standard approach may take a long time to develop and make available to users, and result in limited output, but should result in very user-friendly tools that are likely to enhance care; in contrast, the wiki approach is an inexpensive dissemination method but imposes a greater burden on prospective users to sort through the available options. The grassroots approach, although relying on existing materials, is still labor-intensive, as it requires a thorough review (and sometimes outreach) to collect a comprehensive range of tools. Our approach was similar to the grassroots approach chosen by York and colleagues for their toolkits, with respect to the harvesting of existing tools; however, we differed from the grassroots approach in that we used a more extensive review process to ensure saliency of included tools to our users, reduce redundancy, and ensure a minimum level of quality; in addition, we modified tools to improve their look and feel prior to posting them. As a result, we ended up with a smaller number of tools than has been characteristic of prior VA “grassroots” toolkits [23], which may potentially be compensated for by making it easier for prospective users to find and decide upon their preferred tool. Work by DeWalt and colleagues, who carried out a 2-year toolkit development and testing process for the AHRQ Health Literacy Universal Precautions Toolkit, is perhaps the closest to the gold standard articulated above, with draft tools tested in clinical practice [24] and subsequent practice-based evaluations of individual tools and the toolkit collectively [25–27]. Interestingly, the overall demonstration of the toolkit in clinical practice recommended that technical assistance be provided for tool implementation [25]; CTAC intends to test the value of providing such technical assistance for the care coordination toolkit, in the form of a distance-based coach for half of the project sites, while the other half have access to the toolkit only. Although we did not have the time to test tools in clinical practice during the toolkit development phase, we plan to track which tools are adopted (versus never attempted) by each of the participating sites, and among adopted tools, which tools are successfully implemented and maintained (vs. being abandoned), with follow-up key informant interviews exploring reasons for success in implementation or barriers. In addition, we plan to incorporate users’ feedback to refine the tools in the toolkit as they are used, and add new tools when appropriate. We also intend to evaluate the degree to which the toolkit matches the needs of prospective users, the degree to which users’ readiness to implement a tool affects subsequent success in implementation, and the extent to which the tools affect patient experience of care. We attempted to include tools that have varying implementation difficulty, that have broad applicability, and that have not yet been uniformly adopted in the VA based on our semi-structured interviews. However, it remains to be seen whether the tools will successfully align with clinics’ priorities, and there may be a tradeoff in terms of the difficulty of the tools being implemented and the potential beneficial effect on patient experience of care, with more difficult tools potentially providing more benefit, but also requiring more resources. We realized from the outset that the toolkit can be an adjunct to, but not a replacement for, a serious commitment on the part of clinic leadership and staff to improve care coordination. If successful, the CTAC project will improve care coordination in VA primary care clinics, and will also provide readily applicable methods for spreading improvements throughout the VA. In addition, the project will inform VA policymakers regarding what other implementation strategies, including the use of distance coaching, may influence the use of toolkits. SUPPLEMENTARY MATERIAL Supplementary material is available at Translational Behavioral Medicine online. Compliance with ethical standards Conflict of interest: The authors declare that they have no conflicts of interest. Acknowledgments We thank Michael Ong and Adriana Izquierdo for their help in reviewing the tools as expert clinician reviewers. We also thank Tonya Reznor for programming assistance to deploy the toolkit online, John Øvretveit for helpful comments on a previous version of this manuscript, and individuals who participated in semi-structured interviews for their input. This material is based upon work supported by the Department of Veterans Affairs, Quality Enhancement Research Initiative through a grant to the Care Coordination QUERI Program (QUE 15-276). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government. References 1. Berwick DM , Hackbarth AD . Eliminating waste in US health care . Jama . 2012 ; 307 ( 14 ): 1513 – 1516 . Google Scholar CrossRef Search ADS PubMed 2. Berwick DM , Nolan TW , Whittington J . The triple aim: care, health, and cost . Health Aff (Millwood) . 2008 ; 27 ( 3 ): 759 – 769 . Google Scholar CrossRef Search ADS PubMed 3. Gellad WF . The veterans choice act and dual health system use . J Gen Intern Med . 2016 ; 31 ( 2 ): 153 – 154 . Google Scholar CrossRef Search ADS PubMed 4. Yoon J , Scott JY , Phibbs CS , Wagner TH . Recent trends in veterans affairs chronic condition spending . Popul Health Manag . 2011 ; 14 ( 6 ): 293 – 298 . Google Scholar CrossRef Search ADS PubMed 5. Powell Davies G , Williams AM , Larsen K , Perkins D , Roland M , Harris MF . Coordinating primary health care: an analysis of the outcomes of a systematic review . Med J Aust . 2008 ; 188 ( 8 Suppl ): S65 – S68 . Google Scholar PubMed 6. Lau R , Stevenson F , Ong BN , et al. Achieving change in primary care–causes of the evidence to practice gap: systematic reviews of reviews . Implement Sci . 2016 ; 11 : 40 . Google Scholar CrossRef Search ADS PubMed 7. Powell BJ , Waltz TJ , Chinman MJ , et al. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project . Implement Sci . 2015 ; 10 : 21 . Google Scholar CrossRef Search ADS PubMed 8. Gale RC , Asch SM , Taylor T , et al. The most used and most helpful facilitators for patient-centered medical home implementation . Implement Sci . 2015 ; 10 : 52 . Google Scholar CrossRef Search ADS PubMed 9. McDonald KM , Schultz E , Albin L , et al. Care Coordination Measures Atlas Version 4 . Rockville, MD : Agency for Healthcare Research and Quality ; 2014 . 10. AHRQ Publishing and Communications Guidelines, Section 6: Toolkit Guidance . http://www.ahrq.gov/research/publications/pubcomguide/pcguide6.html Accessibility verified January 4, 2018. 11. Bagalman E. The number of veterans that use VA health care services: a fact sheet . Washington, DC : Congressional Research Service ; 2014 . 12. Chang ET , Wang M , Kirsh S , Rubenstein LV. VA High-Risk Populations in Primary Care . Mineapolis, MN : Accepted for presentation at Academy Health Annual Research Meeting ; 2015 . 13. Rosland AM , Nelson K , Sun H , et al. The patient-centered medical home in the Veterans Health Administration . Am J Manag Care . 2013 ; 19 ( 7 ): e263 – 272 . Google Scholar PubMed 14. Moher D , Liberati A , Tetzlaff J , Altman DG ; PRISMA Group . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Plos Med . 2009 ; 6 ( 7 ): e1000097 . Google Scholar CrossRef Search ADS PubMed 15. Luck J , Bowman C , York L , et al. Multimethod evaluation of the VA’s peer-to-peer Toolkit for patient-centered medical home implementation . J Gen Intern Med . 2014 ; 29 ( Suppl 2 ): S572 – S578 . Google Scholar CrossRef Search ADS PubMed 16. Gale RC , Luck J , York L , Asch S . Peer-to-peer toolkits enhance sharing of best practices across an integrated delivery system . J Patient Cent Res Rev . 2016 ; 3 : 189 . Google Scholar CrossRef Search ADS 17. Ganz DA , Huang C , Saliba D , Shier V , Berlowitz D , Lukas CV , Pelczarski K , Schoelles K , Wallace LC , Neumann P. Preventing Falls in Hospitals: A Toolkit for Improving Quality of Care [online] . In. Rockville, MD : Agency for Healthcare Research and Quality ; 2013 . 18. Engaging Patients in Improving Ambulatory Care: A Compendium of Tools from Maine, Oregon, and Humboldt County, California . https://www.rwjf.org/content/dam/farm/toolkits/toolkits/2013/rwjf404402 Accessibility verified January 4, 2018. 19. Carmichael J , Jassar G , Nguyen PA . Healthcare metrics: Where do pharmacists add value ? Am J Health Syst Pharm . 2016 ; 73 ( 19 ): 1537 – 1547 . Google Scholar CrossRef Search ADS PubMed 20. Parchman ML , Noël PH , Lee S . Primary care attributes, health care system hassles, and chronic illness . Med Care . 2005 ; 43 ( 11 ): 1123 – 1129 . Google Scholar CrossRef Search ADS PubMed 21. Tsan MF , Puglisi T . Health care operations activities that may constitute research: the department of veterans affairs’s perspective . Irb . 2014 ; 36 ( 1 ): 9 – 11 . Google Scholar PubMed 22. Reijers HA. Design and Control of Workflow Processes: Business Process Management for the Service Industry. Chapter 6: Heuristic Workflow Redesign , vol. 2617 . Berlin : Springer ; 2003 . 23. York L , Bruce B , Luck J , et al. Online toolkits for metric-driven quality improvement: the veterans health administration managed grassroots approach . Jt Comm J Qual Patient Saf . 2013 ; 39 ( 12 ): 561 – 569 . Google Scholar CrossRef Search ADS PubMed 24. DeWalt DA , Broucksou KA , Hawk V , et al. Developing and testing the health literacy universal precautions toolkit . Nurs Outlook . 2011 ; 59 ( 2 ): 85 – 94 . Google Scholar CrossRef Search ADS PubMed 25. Mabachi NM , Cifuentes M , Barnard J , et al. Demonstration of the health literacy universal precautions toolkit: lessons for quality improvement . J Ambul Care Manage . 2016 ; 39 ( 3 ): 199 – 208 . Google Scholar CrossRef Search ADS PubMed 26. Weiss BD , Brega AG , LeBlanc WG , et al. Improving the effectiveness of medication review: guidance from the health literacy universal precautions toolkit . J Am Board Fam Med . 2016 ; 29 ( 1 ): 18 – 23 . Google Scholar CrossRef Search ADS PubMed 27. Brega AG , Freedman MA , LeBlanc WG , et al. Using the health literacy universal precautions toolkit to improve the quality of patient materials . J Health Commun . 2015 ; 20 ( Suppl 2 ): 69 – 76 . Google Scholar CrossRef Search ADS PubMed Published by Oxford University Press on behalf of the Society of Behavioral Medicine 2018. This work is written by (a) US Government employee(s) and is in the public domain in the US.

Journal

Translational Behavioral MedicineOxford University Press

Published: May 23, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off