Recruiting primary care practices for practice-based research: a case study of a group-randomized study (TRANSLATE CKD) recruitment process

Recruiting primary care practices for practice-based research: a case study of a group-randomized... Abstract Background We assessed the challenging process of recruiting primary care practices in a practice-based research study. Methods In this descriptive case study of recruitment data collected for a large practice-based study (TRANSLATE CKD), 48 single or multiple-site health care organizations in the USA with a total of 114 practices were invited to participate. We collected quantitative and qualitative measures of recruitment process and outcomes for the first 25 practices recruited. Information about 13 additional practices is not provided due to staff transitions and limited data collection resources. Results Initial outreach was made to 114 practices (from 48 organizations, 41% small); 52 (45%) practices responded with interest. Practices enrolled in the study (n = 25) represented 22% of the total outreach number, or 48% of those initially interested. Average time to enroll was 71 calendar days (range 11–107). There was no difference in the number of days practices remained under recruitment, based on enrolled versus not enrolled (44.8 ± 30.4 versus 46.8 ± 25.4 days, P = 0.86) or by the organization size, i.e. large versus small (defined by having ≤4 distinct practices; 52 ± 23.6 versus 43.6 ± 27.8 days; P = 0.46). The most common recruitment barriers were administrative, e.g. lack of perceived direct organizational benefit, and were more prominent among large organizations. Conclusions Despite the general belief that the research topic, invitation method, and interest in research may facilitate practice recruitment, our results suggest that most of the recruitment challenges represent managerial challenges. Future research projects may need to consider relevant methodologies from businesses administration and marketing fields. PBRN, practice-based research, practice recruitment, primary care, recruitment barriers Introduction There has been a long-recognized need to develop, translate and apply primary care-based research evidence into best patient care practices (1). Provider engagement is critical to carry out research activity and to translate research findings to practice. Unfortunately, only a small percentage of primary care providers report devoting time to research (1), and recruitment of primary care practices to conduct practice-based research remains challenging (1–3). While several studies identified some barriers to primary care physicians’ recruitment and research participation (4,5), to date, evidence is limited to inform provider recruitment planning for primary care research projects (4). While many studies focus on the patient and provider recruitment in clinical trials (6,7), recruitment needs in primary care, however, go far beyond the scope of randomized clinical trials. The research topic, the invitation method and interest in research are important factors for successful recruitment (6). Bower et al. (8) proposed a methodological approach to recruitment strategies for traditional clinical trials, whereby the trial recruitment occurs in four stages (stage 1: professional agrees to participate, stage 2: professional agrees to recruit patients, stage 3: patients agree to enroll, stage 4: patients agree to remain in the study) (8). While the model provides a structured approach to recruitment, Bower’s approach relies on providers being responsible for recruiting patients into clinical trials, which is often not the case for primary care practice-based research. Besides, the recruitment bottleneck typically begins with the first step: recruiting the providers themselves. As described by the Agency for Health care Research and Quality, ‘in order to facilitate and support the research to improve care, practice-based research networks (PBRNs) have emerged as networks of affiliated clinicians in diverse communities’. PBRNs have been viewed as fundamentally necessary to support the translation of research into clinical practice (9,10). Clinicians choose to participate in PBRNs in order to answer questions directly relevant to their practice and to improve the quality of practice and the health of their community. The study presented in this report was conducted in the PBRN setting. Our goal is to present a detailed case study of the recruitment methods and results used in a large practice-based study, TRANSLATE CKD. Specifically, we assessed the recruitment process, compared the outcomes of this process based on key practice characteristics, and explored barriers in recruiting primary care practices in practice-based studies. Methods This is a descriptive case study of primary care practices’ recruitment into a study conducted by the American Academy of Family Physicians National Research Network (AAFP NRN) and the State University of New York at Buffalo, with ethical approval from the Institutional Review Board at both institutions. The TRANSLATE CKD study is a cluster-randomized study aimed to compare models of evidence-based care for patients with chronic kidney disease (CKD) in primary care practices. The detailed study description, objectives and methodology have been reported elsewhere (11). Overall, 48 single or multiple-site United States health care organizations were invited to participate. Here we report the recruitment activities occurring between January, 2012, and October, 2013. The recruitment goal for the TRANSLATE CKD study was 38 practices. Quantitative and qualitative data for recruitment evaluation were collected until the first 25 practices enrolled. The information about the last 13 practices is not provided due to project staff transitions and limited resources to continue systematic data collection. Due to the project administrative decisions, the authors of this work were no longer able to continue systematic data collection related to recruitment process beyond the first 25 practices. Practice inclusion criteria and study expectations Only those practices providing ambulatory primary care as their principal function, located in non-hospital settings, and employing at least one primary care physician, with a minimum of 2000 patients seen in the prior year, were eligible to enroll. ‘Practices’ were defined as distinct office locations that belong to organizations with one or more locations. Candidate practices were drawn from members of the DARTNet Institute (12). The DARTNet Institute is a not-for-profit research institute that coordinates and supports quality improvement activities through the reuse and improved collection of electronic health data and provides an enhanced chronic disease management reminder system. As a part of the study protocol and the practice inclusion criteria, the practice needed to be able to add CKD guideline-based protocols to their chronic disease management reminder system and provide electronic health data. Recruitment procedures We based our recruitment strategy on the best-studied key elements important in facilitating provider recruitment: relevant topic, financial incentive, practice-based research network (PBRN) membership, simple study protocol, low provider time commitment and acceptable practice invitation and recruitment methods (13,14). We originally expected that practice interest in study enrollment would exceed our recruitment target (38 practices total). Recruitment for the study began in January, 2012. The recruitment period covered in this work spans between January 2012 and October 2013. The study reached its recruitment goal of 38 practices in February 2014. As stated above, the data on the last 13 practices recruited between October 2013 and February 2014 are not available and not included in this report. Practices were recruited using a staged recruitment process as follows: Interest generation among practice/organization research representatives (January, 2012) Organization-level engagement (March–July, 2012) Practice-level engagement (April, 2012–October, 2013) Practice follow-up and retention (April 2012–December 2016) End of study (December 2016) To generate interest among practice and health care organization research representatives, we developed a ‘value proposition’ message about the TRANSLATE CKD study (i.e. improve primary care for patients with CKD). We included this message in all recruitment and outreach materials. Additional message reinforcement included a 1-h informational Webinar that was posted online. The recruitment process, stages of recruitment and specific tactics used are described in detail in Figure 1. In addition, the project principal investigator and the project managers followed-up with the practices by phone and e-mail and sent reminder postcards. Figure 1. View largeDownload slide Staged recruitment process overview. Figure 1. View largeDownload slide Staged recruitment process overview. Data collection and analyses We used sequential explanatory mixed methods design to evaluate the recruitment activities. Quantitative data on outcomes of recruitment activities (numbers, rates) were supplemented by the qualitative data obtained through review of key communication (communications, project notes) and project management tracking documents. For categorical variables, counts (n) or percentages (%) are reported as appropriate. Descriptive statistics were computed for continuous variables by the mean and standard deviations. The differences between practices enrolling from multiple-site large health care organizations (defined as having more than four practice locations) and smaller organizations (four or fewer practice locations) were assessed using a Fisher’s Exact Test and independent samples t-test as appropriate with the P value at < 0.05 considered statistically significant. Statistical analyses were conducted using IBM SPSS Statistics 22. Administrative data were used for the analysis, and the qualitative data from notes, communication and recruitment materials were used to explain and demonstrate the findings. Qualitative and mixed methods data were organized using a template style analysis to identify and categorize the units of interest and themes related to the project objectives (15,16). The themes were then broken into sub-themes that were supported by quotations from the interviews (17). The qualitative data collection, analysis and interpretations were conducted by an evaluation team consisting of three members (NL, KA and CS). One member of the evaluation team (KA) joined the project near the end of the study and therefore did not participate in the early data collection activities and was not familiar with project or the practices. This provided an opportunity to conduct objective blinded reviews of the de-identified qualitative data. Qualitative and mixed methods data were organized using a template style analysis to identify and categorize the units of interest and themes related to the project objectives (15,16). The themes were then broken into sub-themes that were supported by quotes from the interviewees (17). Thematic analysis was performed through the process of phases to create established, meaningful patterns and immersion/crystallization analyses were employed when appropriate. These dual processes continued until all data relevant to recruitment barriers were examined, and meaningful patterns and claims that can be well articulated and substantiated emerged. Results We initially contacted 48 health care organizations (comprised of 114 primary care practices total), of which 41% were small organizations (defined by having ≤4 distinct practices). At first, 52 (45%) practices indicated interest. Out of those, 25 (48%) practices enrolled. The 25 enrolled practices represented 22% of total outreach number or 48% of those initially interested. The enrolled practices were from health care organizations of various sizes, having 1–32 distinct practices. The consort diagram showing total number of practices enrolling or dropping out during recruitment is presented in Figure 2. Figure 2. View largeDownload slide Practice recruitment consort diagram. Figure 2. View largeDownload slide Practice recruitment consort diagram. Practices from small health care organizations were around 3.7 times more likely to enroll than practices from larger health care organizations; however, that difference was not statistically significant (OR = 3.67, P = 0.11) (Table 1). Table 1. Likelihood of enrolling in the study based on health care organization size   Total  Enrolled  Did not enroll  Large organization (more than four practice locations)  15  3  12  Small organization (four or fewer practice locations)  33  16  17  Total  48  19a  29a    Total  Enrolled  Did not enroll  Large organization (more than four practice locations)  15  3  12  Small organization (four or fewer practice locations)  33  16  17  Total  48  19a  29a  aFisher’s Exact Test Output: P value = 0.11; odds ratio = 3.67. View Large Among the practices recruited, average total time to enroll from the beginning of recruitment until completion of all required enrollment paperwork was 71 calendar days (range 11–107 days) with 30 days on average for initial acceptance or refusal by the organization and an additional 41 days to complete paperwork. Overall, there was no difference in the number of days practices remained under recruitment based on the outcomes, i.e. enrolled versus not enrolled (44.8 ± 30.4 versus 46.8 ± 25.4 days, P = 0.86) or by the organization size, i.e. large versus small (52 ± 23.6 versus 43.6 ± 27.8 days; P = 0.46). Key recruitment challenges The qualitative data analysis revealed several themes related to key challenges in recruitment process resulting in refusal to participate (see Table 2). We noticed that for single-practice sites the clinical research coordinator is usually a practicing physician, while at health care organizations one clinical research coordinator typically represented all practices within the organization. The clinical coordinators at health care organizations were typically practice, clinic or business managers. In addition, lost paperwork (e.g. practice or organization representative passing forms to clerical staff, who in turn pass to receptionist) was common and could have resulted in recruitment delays. Table 2. Key themes in recruitment challenge and illustrative quotes presented by the practice representatives Recruitment challenge identified  Illustrative feedback received  Decision made on behalf of practices/providers is made by administrators/management who acts as gatekeeper  ‘Several of us have also been discussing what the financial tipping point is for practices/ residency programs to participate in...studies…These studies are so important to the discipline, but what makes sense for the time and energy of the clinicians and entire clinic unit is still a bit unknown…[Our practice] is doing great work in terms of PCMH [Patient Centered Medical Home] and is leading the pack in so many ways—we want to make sure we guide them about what to expect if they participate in the future’.  ‘Once we feel that the practices are comfortable, we will turn them over to you to get the process rolling’.  ‘We are not a good source of practices for this study given the composition and interests of our practice population’.  A need for approval by the board/ organization’s advisory group  ‘We are just so busy so the board decided it is too much and docs will be running screaming from the building if we ask them for one more thing to do’.  Large organizations measure research value/ ‘worthiness’ largely by time, effort and/or money  ‘.. I would like to ask [researcher] if there are any significant …benefits (monetary) for the participants in the CKD study. I understand the value of clinical research, but honestly the clinical staff at our program are so busy that only two things would get us involved, 1. special interest in the topic (which no one here has) or 2. enough money to create special interest. Again, this does not reflect on the inherent value of the study’.  ‘I like your study and I wish we could support it, but I am afraid that we need to decline this offer. … While I believe we could do this, I could not convince our CMOI [Chief Medical Officer] to commit to doing this’.  ‘The study sound intriguing…for us to get providers involved there need to be some carrot’.  Competing demands  ‘I have four current projects I am working on … and I just do not feel I can commit to another right now’.  Recruitment challenge identified  Illustrative feedback received  Decision made on behalf of practices/providers is made by administrators/management who acts as gatekeeper  ‘Several of us have also been discussing what the financial tipping point is for practices/ residency programs to participate in...studies…These studies are so important to the discipline, but what makes sense for the time and energy of the clinicians and entire clinic unit is still a bit unknown…[Our practice] is doing great work in terms of PCMH [Patient Centered Medical Home] and is leading the pack in so many ways—we want to make sure we guide them about what to expect if they participate in the future’.  ‘Once we feel that the practices are comfortable, we will turn them over to you to get the process rolling’.  ‘We are not a good source of practices for this study given the composition and interests of our practice population’.  A need for approval by the board/ organization’s advisory group  ‘We are just so busy so the board decided it is too much and docs will be running screaming from the building if we ask them for one more thing to do’.  Large organizations measure research value/ ‘worthiness’ largely by time, effort and/or money  ‘.. I would like to ask [researcher] if there are any significant …benefits (monetary) for the participants in the CKD study. I understand the value of clinical research, but honestly the clinical staff at our program are so busy that only two things would get us involved, 1. special interest in the topic (which no one here has) or 2. enough money to create special interest. Again, this does not reflect on the inherent value of the study’.  ‘I like your study and I wish we could support it, but I am afraid that we need to decline this offer. … While I believe we could do this, I could not convince our CMOI [Chief Medical Officer] to commit to doing this’.  ‘The study sound intriguing…for us to get providers involved there need to be some carrot’.  Competing demands  ‘I have four current projects I am working on … and I just do not feel I can commit to another right now’.  View Large Discussion We presented a case study conducted in a PBRN setting that provided data on recruitment processes, results, duration and challenges. We originally hypothesized that given the importance of the clinical issue, the multi-modal invitation methods, and recruiting from early-adopter practices, initial practice interest to enroll in the study would exceed our recruitment target. The actual enrollment was about 22% of the practices approached, similar to other reported studies such as Improved Delivery of Cardiovascular Care program (4) that reported recruitment results of 93 practices out of 372, a 25% recruitment rate. Another study reported 24% recruitment rate (6). A recent review reported that recruitment of family physicians in research studies ranged from 19–63% among eligible physicians contacted (3,18). Our own experience and the results of other studies indicate that several factors may contribute to such a variability in the recruitment rates, including the differences in settings, the study topic, protocols and participant requirements, the primary care provider characteristics and experience with research and the recruitment goal (3). Our findings corroborate the published literature on the rather lengthy time required to recruit practices for multi-site practice-based studies. Few other studies reported the recruitment process, with duration ranging from 4 to 10 months (4) and with individual practice recruitment time averaging 19–25 working days (18). Though the practice-based studies rarely report details on recruitment time, based on limited published information available and the findings of our study, the project timeline planning for practice-based research should include a recruitment phase of a sufficient length to achieve practice recruitment target. We found a noticeably higher likelihood of successfully recruiting a practice from smaller health care organizations than from larger organizations. Some recruitment barriers were mostly applicable to large organizations such as the need to obtain study approval from a board of directors. The review by Sahin et al., included only one study on effects of the organization size on recruitment; that study reported that working in larger practices was a facilitator of physician recruitment (19). Whereas, Goodyear-Smith et al., reported that smaller practices took fewer days to recruit while larger practices with employed physicians took the longest (18). Combined, these observations add to the opposing evidence that larger health care organizations may also inhibit practice and provider recruitment, thus indicating need for further research (3). This aspect is important to study as it may affect the practice and provider recruitment approaches as increasing numbers of primary care physicians are employed by larger organizations. With the majority of published studies being conducted on individual provider-level engagement in research and strategies, organization size presents a new characteristic to investigate. According to some, albeit limited, evidence, research studies need to be considered as business propositions especially when recruiting from larger health care organizations. The research team will need to ensure that the objectives of the project align with the organization’s objectives for patient outcomes improvement, resource allocation, and business case. Integration or blending of research with quality improvement (20) with financial considerations to compensate participants for their time (21) or to provide other direct or indirect monetary value may be some of the strategies to demonstrate additional value (‘carrot’) to the practices and organizations. One of the potential incentives (‘carrots’) that is widely acknowledged among providers is an opportunity to obtain continuous medical education (CME) credits. While the study presented in this report offered ABFM MC-FP Part IV with 20 h of CME it is not clear whether this facilitated the practices’ willingness to enroll and remain in the study. More research needs to be done on exploring CME as an effective incentive for practice recruitment and retention. Several studies have suggested solutions to administrative/managerial barriers to research recruitment by adopting methodologies from business world and marketing, including elements such as building brand value, explicit marketing plan, signaling worthiness, making the sale and others (7). Future studies may need to test benefits of these methodologies for recruitment of practices especially from larger organizations. Frameworks for translating business concepts to research can be found elsewhere (7,22–24). The specific eligibility criteria for this particular study may have restricted the recruitment success. Multi-site studies with narrow inclusion criteria may require additional recruitment time, and may require incentives to encourage participation. We reported detailed results only on the first 25 practices, and there could have been differences between the practices that responded first to the call and the practices that made or changed their decision later. Our findings are based on one study, and the results may not be generalizable to other studies or settings. It is important to note that the TRANSLATE CKD study did reach the target recruitment goal, and recruited practices remain in the study to date, but data on the recruitment of the remaining practices are not available. The statistically insignificant differences in our study could be explained by a small sample size, but rigorous studies of recruitment processes similar to this work are lacking to evaluate whether health care organizations size affects recruitment. Conclusion Most of the recruitment challenges presented exemplify managerial challenges. To address these challenges, business and marketing approaches for engaging primary care practices in research should be tested in future studies. Project timelines for practice-based research should include a recruitment phase of a sufficient length to achieve practice recruitment target. Investigators should consider systematic reporting of recruitment efforts and results of recruitment strategies implemented. Declarations Funding: this article is based on TRANSLATE CKD project that is funded by a grant from the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) R01 DK090407. Ethics approval: ethical approval was provided by the Institutional Review Boards at the American Academy of Family Physicians and the State University of New York at Buffalo. The trial is registered with ClinicalTrials.gov, # NCT01767883. Conflict of interest: none. Acknowledgements The authors thank all participants of this project. We would like to acknowledge the AAFP National Research Network for providing essential expertise, staff and support. We thank Ms. Elizabeth Staton of the University of Colorado Department of Family Medicine for providing assistance with editorial revision and manuscript preparation and submission. Ms. Staton is a contract staff member for the National Research Network. Parts of the findings reported in this paper were presented at the Annual Meeting of the North American Primary Care Research Group (NAPCRG), November 2012. References 1. Voorhees JR, Xierali IM, Bazemore AWet al.   A small percentage of family physicians report time devoted to research. J Am Board Fam Med  2013; 26: 7– 8. Google Scholar CrossRef Search ADS PubMed  2. Mold JW, Lipman PD, Durako SJ. Coordinating centers and multi-practice-based research network (PBRN) research. J Am Board Fam Med  2012; 25: 577– 81. Google Scholar CrossRef Search ADS PubMed  3. Sahin D, Yaffe MJ, Sussman T, McCusker J. A mixed studies literature review of family physicians’ participation in research. Fam Med  2014; 46: 503– 14. Google Scholar PubMed  4. Johnston S, Liddy C, Hogg Wet al.   Barriers and facilitators to recruitment of physicians and practices for primary care health services research at one centre. BMC Med Res Methodol  2010; 10: 109. Google Scholar CrossRef Search ADS PubMed  5. Fulda KG, Hahn KA, Young RAet al.   Recruiting Practice-based Research Network (PBRN) physicians to be research participants: lessons learned from the North Texas (NorTex) needs assessment study. J Am Board Fam Med  2011; 24: 610– 5. Google Scholar CrossRef Search ADS PubMed  6. Dormandy E, Kavalier F, Logan Jet al.  ; SHIFT research team. Maximising recruitment and retention of general practices in clinical trials: a case study. Br J Gen Pract  2008; 58: 759– 66. Google Scholar CrossRef Search ADS PubMed  7. Goodarzynejad H, Babamahmoodi A. Project management of randomized clinical trials: a narrative review. Iran Red Crescent Med J  2015; 17: e11602. Google Scholar CrossRef Search ADS PubMed  8. Bower P, Wallace P, Ward Eet al.   Improving recruitment to health research in primary care. Fam Pract  2009; 26: 391– 7. Google Scholar CrossRef Search ADS PubMed  9. Mold JW, Aspy CB, Smith PDet al.   Leveraging practice-based research networks to accelerate implementation and diffusion of chronic kidney disease guidelines in primary care practices: a prospective cohort study. Implement Sci  2014; 9: 169. Google Scholar CrossRef Search ADS PubMed  10. Westfall JM, Mold J, Fagnan L. Practice-based research–“Blue Highways” on the NIH roadmap. JAMA  2007; 297: 403– 6. Google Scholar CrossRef Search ADS PubMed  11. Fox CH, Vest BM, Kahn LSet al.   Improving evidence-based primary care for chronic kidney disease: study protocol for a cluster randomized control trial for translating evidence into practice (TRANSLATE CKD). Implement Sci  2013; 8: 88. Google Scholar CrossRef Search ADS PubMed  12. DARTNet Institute. DARTNet website. http://www.dartnet.info/ (accessed on 5 May 2016). 13. Leathem CS, Cupples ME, Byrne MCet al.   Identifying strategies to maximise recruitment and retention of practices and patients in a multicentre randomised controlled trial of an intervention to optimise secondary prevention for coronary heart disease in primary care. BMC Med Res Methodol  2009; 9: 40. Google Scholar CrossRef Search ADS PubMed  14. Supper I, Ecochard R, Bois Cet al.   How do French GPs consider participating in primary care research: the DRIM study. Fam Pract  2011; 28: 226– 32. Google Scholar CrossRef Search ADS PubMed  15. Cohen DJ, Crabtree BF. Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med  2008; 6: 331– 9. Google Scholar CrossRef Search ADS PubMed  16. Crabtree B, Miller W. A template approach to text analysis: Developing and using codebooks. In: Doing Qualitative Research . Vol 1st. Newbury Park, CA: Sage Publications; 1999: 93– 109. 17. Crabtree BF, Miller WL. A qualitative approach to primary care research: the long interview. Fam Med  1991; 23: 145– 51. Google Scholar PubMed  18. Goodyear-Smith F, York D, Petousis-Harris Het al.   Recruitment of practices in primary care research: the long and the short of it. Fam Pract  2009; 26: 128– 36. Google Scholar CrossRef Search ADS PubMed  19. Jowett SM, Macleod J, Wilson S, Hobbs FD. Research in primary care: extent of involvement and perceived determinants among practitioners from one English region. Br J Gen Pract  2000; 50: 387– 9. Google Scholar PubMed  20. Balasubramanian BA, Cohen DJ, Davis MMet al.   Learning Evaluation: blending quality improvement and implementation research methods to study healthcare innovations. Implement Sci  2015; 10: 31. Google Scholar CrossRef Search ADS PubMed  21. Young RA, Fulda KG, Suzuki Set al.   The influence of research compensation options on Practice-based Research Network (PBRN) physician participation: a North Texas (NorTex) PBRN study. J Am Board Fam Med  2011; 24: 562– 8. Google Scholar CrossRef Search ADS PubMed  22. McDonald AM, Treweek S, Shakur Het al.   Using a business model approach and marketing techniques for recruitment to clinical trials. Trials  2011; 12: 74. Google Scholar CrossRef Search ADS PubMed  23. Francis D, Roberts I, Elbourne DRet al.   Marketing and clinical trials: a case study. Trials  2007; 8: 37. Google Scholar CrossRef Search ADS PubMed  24. Rick J, Graffy J, Knapp Pet al.   Systematic techniques for assisting recruitment to trials (START): study protocol for embedded, randomized controlled trials. Trials  2014; 15: 407. Google Scholar CrossRef Search ADS PubMed  © The Author(s) 2017. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Family Practice Oxford University Press

Recruiting primary care practices for practice-based research: a case study of a group-randomized study (TRANSLATE CKD) recruitment process

Loading next page...
 
/lp/ou_press/recruiting-primary-care-practices-for-practice-based-research-a-case-WiHrVg7WXY
Publisher
Oxford University Press
Copyright
© The Author(s) 2017. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.
ISSN
0263-2136
eISSN
1460-2229
D.O.I.
10.1093/fampra/cmx064
Publisher site
See Article on Publisher Site

Abstract

Abstract Background We assessed the challenging process of recruiting primary care practices in a practice-based research study. Methods In this descriptive case study of recruitment data collected for a large practice-based study (TRANSLATE CKD), 48 single or multiple-site health care organizations in the USA with a total of 114 practices were invited to participate. We collected quantitative and qualitative measures of recruitment process and outcomes for the first 25 practices recruited. Information about 13 additional practices is not provided due to staff transitions and limited data collection resources. Results Initial outreach was made to 114 practices (from 48 organizations, 41% small); 52 (45%) practices responded with interest. Practices enrolled in the study (n = 25) represented 22% of the total outreach number, or 48% of those initially interested. Average time to enroll was 71 calendar days (range 11–107). There was no difference in the number of days practices remained under recruitment, based on enrolled versus not enrolled (44.8 ± 30.4 versus 46.8 ± 25.4 days, P = 0.86) or by the organization size, i.e. large versus small (defined by having ≤4 distinct practices; 52 ± 23.6 versus 43.6 ± 27.8 days; P = 0.46). The most common recruitment barriers were administrative, e.g. lack of perceived direct organizational benefit, and were more prominent among large organizations. Conclusions Despite the general belief that the research topic, invitation method, and interest in research may facilitate practice recruitment, our results suggest that most of the recruitment challenges represent managerial challenges. Future research projects may need to consider relevant methodologies from businesses administration and marketing fields. PBRN, practice-based research, practice recruitment, primary care, recruitment barriers Introduction There has been a long-recognized need to develop, translate and apply primary care-based research evidence into best patient care practices (1). Provider engagement is critical to carry out research activity and to translate research findings to practice. Unfortunately, only a small percentage of primary care providers report devoting time to research (1), and recruitment of primary care practices to conduct practice-based research remains challenging (1–3). While several studies identified some barriers to primary care physicians’ recruitment and research participation (4,5), to date, evidence is limited to inform provider recruitment planning for primary care research projects (4). While many studies focus on the patient and provider recruitment in clinical trials (6,7), recruitment needs in primary care, however, go far beyond the scope of randomized clinical trials. The research topic, the invitation method and interest in research are important factors for successful recruitment (6). Bower et al. (8) proposed a methodological approach to recruitment strategies for traditional clinical trials, whereby the trial recruitment occurs in four stages (stage 1: professional agrees to participate, stage 2: professional agrees to recruit patients, stage 3: patients agree to enroll, stage 4: patients agree to remain in the study) (8). While the model provides a structured approach to recruitment, Bower’s approach relies on providers being responsible for recruiting patients into clinical trials, which is often not the case for primary care practice-based research. Besides, the recruitment bottleneck typically begins with the first step: recruiting the providers themselves. As described by the Agency for Health care Research and Quality, ‘in order to facilitate and support the research to improve care, practice-based research networks (PBRNs) have emerged as networks of affiliated clinicians in diverse communities’. PBRNs have been viewed as fundamentally necessary to support the translation of research into clinical practice (9,10). Clinicians choose to participate in PBRNs in order to answer questions directly relevant to their practice and to improve the quality of practice and the health of their community. The study presented in this report was conducted in the PBRN setting. Our goal is to present a detailed case study of the recruitment methods and results used in a large practice-based study, TRANSLATE CKD. Specifically, we assessed the recruitment process, compared the outcomes of this process based on key practice characteristics, and explored barriers in recruiting primary care practices in practice-based studies. Methods This is a descriptive case study of primary care practices’ recruitment into a study conducted by the American Academy of Family Physicians National Research Network (AAFP NRN) and the State University of New York at Buffalo, with ethical approval from the Institutional Review Board at both institutions. The TRANSLATE CKD study is a cluster-randomized study aimed to compare models of evidence-based care for patients with chronic kidney disease (CKD) in primary care practices. The detailed study description, objectives and methodology have been reported elsewhere (11). Overall, 48 single or multiple-site United States health care organizations were invited to participate. Here we report the recruitment activities occurring between January, 2012, and October, 2013. The recruitment goal for the TRANSLATE CKD study was 38 practices. Quantitative and qualitative data for recruitment evaluation were collected until the first 25 practices enrolled. The information about the last 13 practices is not provided due to project staff transitions and limited resources to continue systematic data collection. Due to the project administrative decisions, the authors of this work were no longer able to continue systematic data collection related to recruitment process beyond the first 25 practices. Practice inclusion criteria and study expectations Only those practices providing ambulatory primary care as their principal function, located in non-hospital settings, and employing at least one primary care physician, with a minimum of 2000 patients seen in the prior year, were eligible to enroll. ‘Practices’ were defined as distinct office locations that belong to organizations with one or more locations. Candidate practices were drawn from members of the DARTNet Institute (12). The DARTNet Institute is a not-for-profit research institute that coordinates and supports quality improvement activities through the reuse and improved collection of electronic health data and provides an enhanced chronic disease management reminder system. As a part of the study protocol and the practice inclusion criteria, the practice needed to be able to add CKD guideline-based protocols to their chronic disease management reminder system and provide electronic health data. Recruitment procedures We based our recruitment strategy on the best-studied key elements important in facilitating provider recruitment: relevant topic, financial incentive, practice-based research network (PBRN) membership, simple study protocol, low provider time commitment and acceptable practice invitation and recruitment methods (13,14). We originally expected that practice interest in study enrollment would exceed our recruitment target (38 practices total). Recruitment for the study began in January, 2012. The recruitment period covered in this work spans between January 2012 and October 2013. The study reached its recruitment goal of 38 practices in February 2014. As stated above, the data on the last 13 practices recruited between October 2013 and February 2014 are not available and not included in this report. Practices were recruited using a staged recruitment process as follows: Interest generation among practice/organization research representatives (January, 2012) Organization-level engagement (March–July, 2012) Practice-level engagement (April, 2012–October, 2013) Practice follow-up and retention (April 2012–December 2016) End of study (December 2016) To generate interest among practice and health care organization research representatives, we developed a ‘value proposition’ message about the TRANSLATE CKD study (i.e. improve primary care for patients with CKD). We included this message in all recruitment and outreach materials. Additional message reinforcement included a 1-h informational Webinar that was posted online. The recruitment process, stages of recruitment and specific tactics used are described in detail in Figure 1. In addition, the project principal investigator and the project managers followed-up with the practices by phone and e-mail and sent reminder postcards. Figure 1. View largeDownload slide Staged recruitment process overview. Figure 1. View largeDownload slide Staged recruitment process overview. Data collection and analyses We used sequential explanatory mixed methods design to evaluate the recruitment activities. Quantitative data on outcomes of recruitment activities (numbers, rates) were supplemented by the qualitative data obtained through review of key communication (communications, project notes) and project management tracking documents. For categorical variables, counts (n) or percentages (%) are reported as appropriate. Descriptive statistics were computed for continuous variables by the mean and standard deviations. The differences between practices enrolling from multiple-site large health care organizations (defined as having more than four practice locations) and smaller organizations (four or fewer practice locations) were assessed using a Fisher’s Exact Test and independent samples t-test as appropriate with the P value at < 0.05 considered statistically significant. Statistical analyses were conducted using IBM SPSS Statistics 22. Administrative data were used for the analysis, and the qualitative data from notes, communication and recruitment materials were used to explain and demonstrate the findings. Qualitative and mixed methods data were organized using a template style analysis to identify and categorize the units of interest and themes related to the project objectives (15,16). The themes were then broken into sub-themes that were supported by quotations from the interviews (17). The qualitative data collection, analysis and interpretations were conducted by an evaluation team consisting of three members (NL, KA and CS). One member of the evaluation team (KA) joined the project near the end of the study and therefore did not participate in the early data collection activities and was not familiar with project or the practices. This provided an opportunity to conduct objective blinded reviews of the de-identified qualitative data. Qualitative and mixed methods data were organized using a template style analysis to identify and categorize the units of interest and themes related to the project objectives (15,16). The themes were then broken into sub-themes that were supported by quotes from the interviewees (17). Thematic analysis was performed through the process of phases to create established, meaningful patterns and immersion/crystallization analyses were employed when appropriate. These dual processes continued until all data relevant to recruitment barriers were examined, and meaningful patterns and claims that can be well articulated and substantiated emerged. Results We initially contacted 48 health care organizations (comprised of 114 primary care practices total), of which 41% were small organizations (defined by having ≤4 distinct practices). At first, 52 (45%) practices indicated interest. Out of those, 25 (48%) practices enrolled. The 25 enrolled practices represented 22% of total outreach number or 48% of those initially interested. The enrolled practices were from health care organizations of various sizes, having 1–32 distinct practices. The consort diagram showing total number of practices enrolling or dropping out during recruitment is presented in Figure 2. Figure 2. View largeDownload slide Practice recruitment consort diagram. Figure 2. View largeDownload slide Practice recruitment consort diagram. Practices from small health care organizations were around 3.7 times more likely to enroll than practices from larger health care organizations; however, that difference was not statistically significant (OR = 3.67, P = 0.11) (Table 1). Table 1. Likelihood of enrolling in the study based on health care organization size   Total  Enrolled  Did not enroll  Large organization (more than four practice locations)  15  3  12  Small organization (four or fewer practice locations)  33  16  17  Total  48  19a  29a    Total  Enrolled  Did not enroll  Large organization (more than four practice locations)  15  3  12  Small organization (four or fewer practice locations)  33  16  17  Total  48  19a  29a  aFisher’s Exact Test Output: P value = 0.11; odds ratio = 3.67. View Large Among the practices recruited, average total time to enroll from the beginning of recruitment until completion of all required enrollment paperwork was 71 calendar days (range 11–107 days) with 30 days on average for initial acceptance or refusal by the organization and an additional 41 days to complete paperwork. Overall, there was no difference in the number of days practices remained under recruitment based on the outcomes, i.e. enrolled versus not enrolled (44.8 ± 30.4 versus 46.8 ± 25.4 days, P = 0.86) or by the organization size, i.e. large versus small (52 ± 23.6 versus 43.6 ± 27.8 days; P = 0.46). Key recruitment challenges The qualitative data analysis revealed several themes related to key challenges in recruitment process resulting in refusal to participate (see Table 2). We noticed that for single-practice sites the clinical research coordinator is usually a practicing physician, while at health care organizations one clinical research coordinator typically represented all practices within the organization. The clinical coordinators at health care organizations were typically practice, clinic or business managers. In addition, lost paperwork (e.g. practice or organization representative passing forms to clerical staff, who in turn pass to receptionist) was common and could have resulted in recruitment delays. Table 2. Key themes in recruitment challenge and illustrative quotes presented by the practice representatives Recruitment challenge identified  Illustrative feedback received  Decision made on behalf of practices/providers is made by administrators/management who acts as gatekeeper  ‘Several of us have also been discussing what the financial tipping point is for practices/ residency programs to participate in...studies…These studies are so important to the discipline, but what makes sense for the time and energy of the clinicians and entire clinic unit is still a bit unknown…[Our practice] is doing great work in terms of PCMH [Patient Centered Medical Home] and is leading the pack in so many ways—we want to make sure we guide them about what to expect if they participate in the future’.  ‘Once we feel that the practices are comfortable, we will turn them over to you to get the process rolling’.  ‘We are not a good source of practices for this study given the composition and interests of our practice population’.  A need for approval by the board/ organization’s advisory group  ‘We are just so busy so the board decided it is too much and docs will be running screaming from the building if we ask them for one more thing to do’.  Large organizations measure research value/ ‘worthiness’ largely by time, effort and/or money  ‘.. I would like to ask [researcher] if there are any significant …benefits (monetary) for the participants in the CKD study. I understand the value of clinical research, but honestly the clinical staff at our program are so busy that only two things would get us involved, 1. special interest in the topic (which no one here has) or 2. enough money to create special interest. Again, this does not reflect on the inherent value of the study’.  ‘I like your study and I wish we could support it, but I am afraid that we need to decline this offer. … While I believe we could do this, I could not convince our CMOI [Chief Medical Officer] to commit to doing this’.  ‘The study sound intriguing…for us to get providers involved there need to be some carrot’.  Competing demands  ‘I have four current projects I am working on … and I just do not feel I can commit to another right now’.  Recruitment challenge identified  Illustrative feedback received  Decision made on behalf of practices/providers is made by administrators/management who acts as gatekeeper  ‘Several of us have also been discussing what the financial tipping point is for practices/ residency programs to participate in...studies…These studies are so important to the discipline, but what makes sense for the time and energy of the clinicians and entire clinic unit is still a bit unknown…[Our practice] is doing great work in terms of PCMH [Patient Centered Medical Home] and is leading the pack in so many ways—we want to make sure we guide them about what to expect if they participate in the future’.  ‘Once we feel that the practices are comfortable, we will turn them over to you to get the process rolling’.  ‘We are not a good source of practices for this study given the composition and interests of our practice population’.  A need for approval by the board/ organization’s advisory group  ‘We are just so busy so the board decided it is too much and docs will be running screaming from the building if we ask them for one more thing to do’.  Large organizations measure research value/ ‘worthiness’ largely by time, effort and/or money  ‘.. I would like to ask [researcher] if there are any significant …benefits (monetary) for the participants in the CKD study. I understand the value of clinical research, but honestly the clinical staff at our program are so busy that only two things would get us involved, 1. special interest in the topic (which no one here has) or 2. enough money to create special interest. Again, this does not reflect on the inherent value of the study’.  ‘I like your study and I wish we could support it, but I am afraid that we need to decline this offer. … While I believe we could do this, I could not convince our CMOI [Chief Medical Officer] to commit to doing this’.  ‘The study sound intriguing…for us to get providers involved there need to be some carrot’.  Competing demands  ‘I have four current projects I am working on … and I just do not feel I can commit to another right now’.  View Large Discussion We presented a case study conducted in a PBRN setting that provided data on recruitment processes, results, duration and challenges. We originally hypothesized that given the importance of the clinical issue, the multi-modal invitation methods, and recruiting from early-adopter practices, initial practice interest to enroll in the study would exceed our recruitment target. The actual enrollment was about 22% of the practices approached, similar to other reported studies such as Improved Delivery of Cardiovascular Care program (4) that reported recruitment results of 93 practices out of 372, a 25% recruitment rate. Another study reported 24% recruitment rate (6). A recent review reported that recruitment of family physicians in research studies ranged from 19–63% among eligible physicians contacted (3,18). Our own experience and the results of other studies indicate that several factors may contribute to such a variability in the recruitment rates, including the differences in settings, the study topic, protocols and participant requirements, the primary care provider characteristics and experience with research and the recruitment goal (3). Our findings corroborate the published literature on the rather lengthy time required to recruit practices for multi-site practice-based studies. Few other studies reported the recruitment process, with duration ranging from 4 to 10 months (4) and with individual practice recruitment time averaging 19–25 working days (18). Though the practice-based studies rarely report details on recruitment time, based on limited published information available and the findings of our study, the project timeline planning for practice-based research should include a recruitment phase of a sufficient length to achieve practice recruitment target. We found a noticeably higher likelihood of successfully recruiting a practice from smaller health care organizations than from larger organizations. Some recruitment barriers were mostly applicable to large organizations such as the need to obtain study approval from a board of directors. The review by Sahin et al., included only one study on effects of the organization size on recruitment; that study reported that working in larger practices was a facilitator of physician recruitment (19). Whereas, Goodyear-Smith et al., reported that smaller practices took fewer days to recruit while larger practices with employed physicians took the longest (18). Combined, these observations add to the opposing evidence that larger health care organizations may also inhibit practice and provider recruitment, thus indicating need for further research (3). This aspect is important to study as it may affect the practice and provider recruitment approaches as increasing numbers of primary care physicians are employed by larger organizations. With the majority of published studies being conducted on individual provider-level engagement in research and strategies, organization size presents a new characteristic to investigate. According to some, albeit limited, evidence, research studies need to be considered as business propositions especially when recruiting from larger health care organizations. The research team will need to ensure that the objectives of the project align with the organization’s objectives for patient outcomes improvement, resource allocation, and business case. Integration or blending of research with quality improvement (20) with financial considerations to compensate participants for their time (21) or to provide other direct or indirect monetary value may be some of the strategies to demonstrate additional value (‘carrot’) to the practices and organizations. One of the potential incentives (‘carrots’) that is widely acknowledged among providers is an opportunity to obtain continuous medical education (CME) credits. While the study presented in this report offered ABFM MC-FP Part IV with 20 h of CME it is not clear whether this facilitated the practices’ willingness to enroll and remain in the study. More research needs to be done on exploring CME as an effective incentive for practice recruitment and retention. Several studies have suggested solutions to administrative/managerial barriers to research recruitment by adopting methodologies from business world and marketing, including elements such as building brand value, explicit marketing plan, signaling worthiness, making the sale and others (7). Future studies may need to test benefits of these methodologies for recruitment of practices especially from larger organizations. Frameworks for translating business concepts to research can be found elsewhere (7,22–24). The specific eligibility criteria for this particular study may have restricted the recruitment success. Multi-site studies with narrow inclusion criteria may require additional recruitment time, and may require incentives to encourage participation. We reported detailed results only on the first 25 practices, and there could have been differences between the practices that responded first to the call and the practices that made or changed their decision later. Our findings are based on one study, and the results may not be generalizable to other studies or settings. It is important to note that the TRANSLATE CKD study did reach the target recruitment goal, and recruited practices remain in the study to date, but data on the recruitment of the remaining practices are not available. The statistically insignificant differences in our study could be explained by a small sample size, but rigorous studies of recruitment processes similar to this work are lacking to evaluate whether health care organizations size affects recruitment. Conclusion Most of the recruitment challenges presented exemplify managerial challenges. To address these challenges, business and marketing approaches for engaging primary care practices in research should be tested in future studies. Project timelines for practice-based research should include a recruitment phase of a sufficient length to achieve practice recruitment target. Investigators should consider systematic reporting of recruitment efforts and results of recruitment strategies implemented. Declarations Funding: this article is based on TRANSLATE CKD project that is funded by a grant from the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) R01 DK090407. Ethics approval: ethical approval was provided by the Institutional Review Boards at the American Academy of Family Physicians and the State University of New York at Buffalo. The trial is registered with ClinicalTrials.gov, # NCT01767883. Conflict of interest: none. Acknowledgements The authors thank all participants of this project. We would like to acknowledge the AAFP National Research Network for providing essential expertise, staff and support. We thank Ms. Elizabeth Staton of the University of Colorado Department of Family Medicine for providing assistance with editorial revision and manuscript preparation and submission. Ms. Staton is a contract staff member for the National Research Network. Parts of the findings reported in this paper were presented at the Annual Meeting of the North American Primary Care Research Group (NAPCRG), November 2012. References 1. Voorhees JR, Xierali IM, Bazemore AWet al.   A small percentage of family physicians report time devoted to research. J Am Board Fam Med  2013; 26: 7– 8. Google Scholar CrossRef Search ADS PubMed  2. Mold JW, Lipman PD, Durako SJ. Coordinating centers and multi-practice-based research network (PBRN) research. J Am Board Fam Med  2012; 25: 577– 81. Google Scholar CrossRef Search ADS PubMed  3. Sahin D, Yaffe MJ, Sussman T, McCusker J. A mixed studies literature review of family physicians’ participation in research. Fam Med  2014; 46: 503– 14. Google Scholar PubMed  4. Johnston S, Liddy C, Hogg Wet al.   Barriers and facilitators to recruitment of physicians and practices for primary care health services research at one centre. BMC Med Res Methodol  2010; 10: 109. Google Scholar CrossRef Search ADS PubMed  5. Fulda KG, Hahn KA, Young RAet al.   Recruiting Practice-based Research Network (PBRN) physicians to be research participants: lessons learned from the North Texas (NorTex) needs assessment study. J Am Board Fam Med  2011; 24: 610– 5. Google Scholar CrossRef Search ADS PubMed  6. Dormandy E, Kavalier F, Logan Jet al.  ; SHIFT research team. Maximising recruitment and retention of general practices in clinical trials: a case study. Br J Gen Pract  2008; 58: 759– 66. Google Scholar CrossRef Search ADS PubMed  7. Goodarzynejad H, Babamahmoodi A. Project management of randomized clinical trials: a narrative review. Iran Red Crescent Med J  2015; 17: e11602. Google Scholar CrossRef Search ADS PubMed  8. Bower P, Wallace P, Ward Eet al.   Improving recruitment to health research in primary care. Fam Pract  2009; 26: 391– 7. Google Scholar CrossRef Search ADS PubMed  9. Mold JW, Aspy CB, Smith PDet al.   Leveraging practice-based research networks to accelerate implementation and diffusion of chronic kidney disease guidelines in primary care practices: a prospective cohort study. Implement Sci  2014; 9: 169. Google Scholar CrossRef Search ADS PubMed  10. Westfall JM, Mold J, Fagnan L. Practice-based research–“Blue Highways” on the NIH roadmap. JAMA  2007; 297: 403– 6. Google Scholar CrossRef Search ADS PubMed  11. Fox CH, Vest BM, Kahn LSet al.   Improving evidence-based primary care for chronic kidney disease: study protocol for a cluster randomized control trial for translating evidence into practice (TRANSLATE CKD). Implement Sci  2013; 8: 88. Google Scholar CrossRef Search ADS PubMed  12. DARTNet Institute. DARTNet website. http://www.dartnet.info/ (accessed on 5 May 2016). 13. Leathem CS, Cupples ME, Byrne MCet al.   Identifying strategies to maximise recruitment and retention of practices and patients in a multicentre randomised controlled trial of an intervention to optimise secondary prevention for coronary heart disease in primary care. BMC Med Res Methodol  2009; 9: 40. Google Scholar CrossRef Search ADS PubMed  14. Supper I, Ecochard R, Bois Cet al.   How do French GPs consider participating in primary care research: the DRIM study. Fam Pract  2011; 28: 226– 32. Google Scholar CrossRef Search ADS PubMed  15. Cohen DJ, Crabtree BF. Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med  2008; 6: 331– 9. Google Scholar CrossRef Search ADS PubMed  16. Crabtree B, Miller W. A template approach to text analysis: Developing and using codebooks. In: Doing Qualitative Research . Vol 1st. Newbury Park, CA: Sage Publications; 1999: 93– 109. 17. Crabtree BF, Miller WL. A qualitative approach to primary care research: the long interview. Fam Med  1991; 23: 145– 51. Google Scholar PubMed  18. Goodyear-Smith F, York D, Petousis-Harris Het al.   Recruitment of practices in primary care research: the long and the short of it. Fam Pract  2009; 26: 128– 36. Google Scholar CrossRef Search ADS PubMed  19. Jowett SM, Macleod J, Wilson S, Hobbs FD. Research in primary care: extent of involvement and perceived determinants among practitioners from one English region. Br J Gen Pract  2000; 50: 387– 9. Google Scholar PubMed  20. Balasubramanian BA, Cohen DJ, Davis MMet al.   Learning Evaluation: blending quality improvement and implementation research methods to study healthcare innovations. Implement Sci  2015; 10: 31. Google Scholar CrossRef Search ADS PubMed  21. Young RA, Fulda KG, Suzuki Set al.   The influence of research compensation options on Practice-based Research Network (PBRN) physician participation: a North Texas (NorTex) PBRN study. J Am Board Fam Med  2011; 24: 562– 8. Google Scholar CrossRef Search ADS PubMed  22. McDonald AM, Treweek S, Shakur Het al.   Using a business model approach and marketing techniques for recruitment to clinical trials. Trials  2011; 12: 74. Google Scholar CrossRef Search ADS PubMed  23. Francis D, Roberts I, Elbourne DRet al.   Marketing and clinical trials: a case study. Trials  2007; 8: 37. Google Scholar CrossRef Search ADS PubMed  24. Rick J, Graffy J, Knapp Pet al.   Systematic techniques for assisting recruitment to trials (START): study protocol for embedded, randomized controlled trials. Trials  2014; 15: 407. Google Scholar CrossRef Search ADS PubMed  © The Author(s) 2017. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.

Journal

Family PracticeOxford University Press

Published: Feb 1, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off