Background: Post-market surveillance of medical devices is reliant on physician reporting of adverse medical device events (AMDEs). Few studies have examined factors that influence whether and how physicians report AMDEs, an essential step in the development of behaviour change interventions. This study was a secondary analysis comparing application of the Theoretical Domains Framework (TDF) and the Tailored Implementation for Chronic Diseases (TICD) framework to identify potential behaviour change interventions that correspond to determinants of AMDE reporting. Methods: A previous study involving qualitative interviews with Canadian physicians that implant medical devices identified themes reflecting AMDE reporting determinants. In this secondary analysis, themes that emerged from the primary analysis were independently mapped to the TDF and TICD. Determinants and corresponding intervention options arising from both frameworks (and both mappers) were compared. Results: Both theoretical frameworks were useful for identifying interventions corresponding to behavioural determinants of AMDE reporting. Information or education strategies that provide evidence about AMDEs, and audit and feedback of AMDE data were identified as interventions to target the theme of physician beliefs; improving information systems, and reminder cues, prompts and awards were identified as interventions to address determinants arising from the organization or systems themes; and modifying financial/non-financial incentives and sharing data on outcomes associated with AMDEs were identified as interventions to target device market themes. Numerous operational challenges were encountered in the application of both frameworks including a lack of clarity about how directly relevant to themes the domains/determinants should be, how many domains/determinants to select, if and how to resolve discrepancies across multiple mappers, and how to choose interventions from among the large number associated with selected domains/determinants. Conclusions: Given discrepancies in mapping themes to determinants/domains and the resulting interventions offered by the two frameworks, uncertainty remains about how to choose interventions that best match behavioural determinants in a given context. Further research is needed to provide more nuanced guidance on the application of TDF and TICD for a broader audience, which is likely to increase the utility and uptake of these frameworks in practice. Keywords: Equipment and supplies, Physicians’ practice patterns, Determinants, Medical errors, Reporting, Qualitative research * Correspondence: firstname.lastname@example.org University Health Network, Toronto, Canada Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 2 of 14 Background To learn about AMDE reporting behaviour, we inter- A growing body of research in implementation science viewed 22 Canadian physicians who varied by geograph- has employed classic or implementation science theories ical region and career stage; 10 implanted cardiovascular or theoretical frameworks to investigate behavioural devices and 12 implanted orthopedic devices . When determinants influencing the use of evidence-based in- AMDEs arose, they often developed work-around solu- novations by health care professionals . Given the tions to continue using the same type of device, or they undesirable prevalence of over-, under- or misuse of in- chose to use other comparable devices available on the novations and their inconsistent impact on patient out- market. Some participants said they informally shared comes , systematic categorization of determinants has information about AMDEs with colleagues or industry been highlighted as a strategy to inform the selection of representatives, however most did not. Determinants of interventions that best mitigate or address those deter- AMDE reporting were identified at the level of the phys- minants. The Theoretical Domains Framework (TDF) ician (i.e. beliefs about adverse events, device prefer-  and the Tailored Implementation for Chronic Dis- ences); organization or system (i.e. lack of hospital, eases (TICD) checklist  are two prominent, validated national or international reporting policies, systems or theoretical frameworks that were rigorously developed incentives); and the device market (i.e. purchasing group based on review of the literature followed by inter- contract obligations) . national expert consensus. Both facilitate the design of As invasive health care technologies, the characteris- implementation strategies by identifying one or more in- tics and uses of higher-risk medical devices differ from terventions that may be appropriate for addressing be- those of other innovations such as practice guidelines, havioural determinants. clinical procedures, or quality improvement processes or Unfortunately, application of these theoretical frame- tools. Hence, determinants of their use may also differ, works to develop and implement change strategies has providing a unique context within which to study the proven challenging , with an inconsistent impact on application of theoretical frameworks for selecting be- health care delivery or patient outcomes . There is a havioural interventions. The purpose of this study was need to improve the selection of behavioural interventions to (1) categorize determinants of AMDE reporting be- so that they reliably lead to health care improvement. haviour that emerged in the primary study using the Hence, more insight is needed about the similarities and TDF and TICD; (2) systematically identify interventions differences in the content and application of commonly that could promote and support AMDE reporting; and used theoretical frameworks to understand how their use (3) compare the determinants and interventions identi- can be optimized when choosing and designing behaviour fied by the TDF and TICD as a means of exploring how change strategies. to optimize the use of those theoretical frameworks in Previous research has focused on the determinants of behavioural intervention design. At a practical level, implementing practice guidelines, clinical tests or proce- study results will identify interventions that are likely to dures, and quality improvement processes or tools [7, 8]. improve AMDE reporting, thereby optimizing the use Despite widespread use of medical devices, little atten- and outcomes of higher-risk medical devices. Simultan- tion has been devoted to understanding determinants of eously, this work will contribute to the implementation the reporting of adverse events associated with their use. science literature by broadening our understanding of Medical devices include a wide range of health or med- the relevance and application of theoretical frameworks ical instruments essential for the prevention, diagnosis, in identifying or describing determinants of innovation cure or management of a disease or abnormal physical use, and selecting corresponding behavioural interven- condition . Those considered higher risk for adverse tions for change. medical device events (AMDEs) include orthopedic im- plants such as hip or knee joints and cardiovascular im- Methods plants such as pacemakers or implantable cardioverter Study design defibrillators [10, 11]. AMDEs may result from limita- AMDE reporting determinants were mapped to the TDF tions in device design or function, and account for 10% and TICD to compare determinant domains, determi- of patient safety incidents in hospitals . Growing nants and corresponding recommended behavioural inter- concern about AMDEs has led to calls for greater moni- ventions. The two authors (LD and ARG) independently toring of outcomes associated with their use . How- mapped the determinants using each framework. LD is an ever, registries are not present in every jurisdiction or implementation scientist with experience in studying the for every type of medical device. In the absence of sys- determinants of physician behavior as it relates to pre- tematic data collection, the identification and sharing of scribing practices , the interdisciplinary management information about AMDEs relies on voluntary reporting of residents in long-term care , and the determinants by physicians. of patient adherence to recommended treatment following Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 3 of 14 a myocardial infarction . ARG is an implementation of physician beliefs; policies, processes, and systems; and scientist with extensive experience in studying determi- the device market . nants of the use of innovations including teamwork in cancer diagnostic assessment programs , timely triage Data mapping and referral of trauma patients , the surgical safety Mapping of AMDE reporting determinants to the TDF checklist , guidelines  and integrated knowledge and TICD was independently performed by LD and translation . ARG has also evaluated the use of theory ARG. To do this, both used the same version of the TDF in assessing barriers of innovation use  and in plan-  and TICD  instruments that listed determinant ning behavioural interventions to implement guidelines domains, individual determinants (for TICD), and corre- . ARG had employed the TICD to collect or analyze sponding behavioural interventions. The intent was to data in previous studies; she was familiar with the TDF undertake naturalistic application of the TDF and TICD but had not applied it in previous work. LD had not previ- that relied solely on the content and guidance provided ously applied the TICD but had previous training and ex- by the theoretical frameworks themselves. LD and ARG perience related to the TDF. This study was based on did not review or discuss the content of the TDF or secondary analysis of qualitative data and did not require TICD before the independent mapping exercise, nor did ethics approval. However, the University Health Network they attempt to resolve and reach consensus on discrep- Research Ethics Board provided ethical approval for the ancies after mapping. This was an intentional methodo- qualitative study that generated data upon which this logical decision to facilitate comparison across mappers study is based, and participants of the qualitative study using only the frameworks themselves as a guide. AMDE had provided written informed consent prior to being reporting determinants were matched to determinant interviewed . domains or individual determinants by reading the defi- nitions and examples provided in each framework. LD and ARG each generated a table in which AMDE report- Implementation frameworks ing themes and exemplar quotes were listed along with The TDF includes 84 individual determinants across 14 TDF and TICD domains or determinants thought to be domains (knowledge, skills, social or professional role relevant and reflective of the data. and identity, beliefs about capabilities, optimism, beliefs about consequences, reinforcement, intentions, goals; Data analysis memory, attention and decision processes; environment, The two tables reflecting independent mapping were context and resources, social influences, emotion, behav- collated to illustrate the TDF and TICD domains or de- ioural regulation). These domains, and not the individual terminants selected by both LD and ARG, and by LD determinants within them, are linked with 93 behav- alone and ARG alone. Behavioural interventions corre- ioural interventions (referred to as behaviour change sponding to each domain or individual determinant were techniques) across 16 overarching categories . The extracted from the TDF and TICD and added to the col- TICD includes 57 individual determinants grouped in 7 lated table. Domains, determinants and corresponding domains (guideline factors, individual health professional interventions identified by LD and ARG in the TDF and factors, patient factors, professional interactions, incen- TICD were enumerated and compared. tives and resources, capacity for organizational change; social, political, and legal factors), and links individual Results determinants with one or more of 116 behavioural inter- Mapping of AMDE reporting themes to TDF and TICD ventions . Table 1 summarizes the TDF domains and Table 2 sum- marizes the TICD determinants selected by one or both Data collection mappers. AMDE reporting determinants and exemplar quotes that illustrated determinants were acquired from the previ- All themes were successfully mapped to both frameworks ously conducted study (Additional file 1). Methods All AMDE reporting themes (noted in italics throughout for the previous study are published elsewhere . In the manuscript) were directly and clearly addressed by brief, qualitative interviews with physicians that im- both frameworks, and therefore mapped to one or more planted cardiovascular and orthopedic implants were TDF domain and TICD determinant. For example, the conducted by ARG. Themes reflecting determinants theme ‘AMDEs were considered unexpected or unavoid- were generated, reviewed and discussed by the entire able’ aligned with the TDF domain of ‘Beliefs about con- eight-person research team on four separate occasions sequences’ and the theme ‘Lack of responsiveness to to assess thematic saturation, agree upon themes, and AMDEs from industry’ was readily mapped to the TDF interpret data. Themes were organized in the categories domain of ‘Reinforcement’. Similarly, the theme ‘No Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 4 of 14 Table 1 Comparison of TDF determinant mapping across mappers Determinant themes from AMDE study TDF domains selected by mappers LD ARG Domain Match PHYSICIAN BELIEFS AMDEs considered expected or unavoidable and not Beliefs about consequences Beliefs about consequences Yes adverse unless outcomes catastrophic; viewed as Social-professional role and identity – No more severe in other specialties AMDEs within 2 years of use were considered unusual Beliefs about consequences Beliefs about consequences Yes Views about cause of AMDEs confounded by multiple Beliefs about consequences – No factors – Knowledge No Incidence of AMDEs has decreased, thus devices Beliefs about consequences – No were thought to be improved – Optimism No Sub-total unique or matching domains 2 3 2/7 (28.6%) POLICIES, PROCESSES or SYSTEMS Follow-up of device-related outcomes beyond short- Environmental context and resources Environmental context and resources Yes term results done elsewhere Social-professional role and identity – No Devices implanted not recorded in patient records Environmental context and resources Environmental context and resources Yes No hospital, national or international systems for Environmental context and resources Environmental context and resources Yes AMDE reporting – Reinforcement No Knowledge – No Behavioural regulation – No Sub-total unique or matching domains 4 2 3/7 (42.9%) DEVICE MARKET Use of specific devices often determined by Environmental context and resources Environmental context and resources Yes purchase group contract obligations Lack of responsiveness to AMDEs from industry Reinforcement Reinforcement Yes Knowledge – No Optimism – No Beliefs about consequences – No – Environmental context and resources No Sub-total unique or matching domains 5 2 2/6 (33.3%) Total unique or matching domains 6 5 7/20 (35.0%) hospital, national or international systems for AMDE that AMDE reporting themes often mapped to more than reporting’ was readily mapped to the TICD determinant one domain or determinant. For example, the theme ‘No ‘Incentives and resources: information system’ and ‘Use hospital, national or international systems for AMDE of specific devices often determined by purchasing group reporting’ mapped to 4 different TDF domains (Environ- contracts obligations’ was mapped to the TICD deter- mental context and resources, Reinforcement, Knowledge, minant ‘Health professional behaviour: capacity to plan and Behavioural regulation). The same theme mapped to change’. 5 different TICD domains, representing 9 unique determi- nants [Incentives and resources (4 determinants): infor- A range of domains and determinants were identified mation system, availability of necessary resources, AMDE reporting determinants were mapped to multiple non-financial incentives and disincentives, and quality domains and determinants, revealing the interplay of assurance and patient safety systems; Capacity for multi-level determinants that influence AMDE reporting, organizational change (2 determinants): regulations, rules, in addition to the complexity of applying the TDF and and policies, and monitoring and feedback; Health profes- TICD. In part this was because the previous study  sional knowledge and skills (1 determinant): domain identified that physician, organizational, system, and mar- knowledge; Health professional cognitions (1 determin- ket level factors influenced whether and how physicians ant): intention and motivation; Health professional behav- reported AMDEs. This was compounded by the reality iour (1 determinant): self-monitoring or feedback]. Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 5 of 14 Table 2 Comparison of TICD determinant mapping across mappers Determinant themes from AMDE study TICD domains:determinants selected by mappers LD ARG Determinant Match PHYSICIAN BELIEFS AMDEs considered expected or unavoidable Health professional cognitions: expected Health professional cognitions: expected Yes and not adverse unless outcomes catastrophic; outcome outcome viewed as more severe in other specialties Health professional cognitions: – No agreement with the recommendation AMDEs within 2 years of use were considered Health professional cognitions: expected Health professional cognitions: expected Yes unusual outcome outcome Health professional cognitions: – No agreement with recommendations Views about cause of AMDEs confounded by Health professional cognitions: – No multiple factors agreement with the recommendation – Health professional knowledge and No skills: domain knowledge Incidence of AMDEs has decreased, thus Health professional cognitions: expected Health professional cognitions: expected Yes devices were thought to be improved outcome outcome Health professional cognitions: – No agreement with recommendations Sub-total unique or matching determinants 3 2 3/8 (37.5%) POLICIES, PROCESSES or SYSTEMS Follow-up of device-related outcomes beyond Recommended behaviour: observability Recommended behaviour: observability Yes short-term results done elsewhere Health professional cognitions: intention – No and motivation Health professional behaviour: nature of – No the behaviour – Health professional knowledge and No skills: knowledge about own practice – Health professional behaviour: self- No monitoring or feedback – Professional interactions: referral No processes Devices implanted not recorded in patient Incentives and resources: information Incentives and resources: information Yes records system system – Health professional knowledge and No skills: knowledge about own practice – Health professional behaviour: capacity No to plan change – Health professional behaviour: self- No monitoring or feedback No hospital, national or international systems Incentives and resources: information Incentives and resources: information Yes for AMDE reporting system system Incentives and resources: availability of Incentives and resources: availability of Yes necessary resources necessary resources Capacity for organizational change: Capacity for organizational change: Yes regulations, rules and policies regulations, rules and policies Health professional knowledge and – No skills: domain knowledge – Health professional cognitions: intention No and motivation – Health professional behaviour: self- No monitoring or feedback Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 6 of 14 Table 2 Comparison of TICD determinant mapping across mappers (Continued) Determinant themes from AMDE study TICD domains:determinants selected by mappers LD ARG Determinant Match – Incentives and resources: non-financial No incentives and disincentives – Incentives and resources: quality No assurance and patient safety systems – Capacity for organization change: No monitoring and feedback Sub-total unique or matching determinants 7 11 5/19 (26.3%) DEVICE MARKET Use of specific devices often determined by Health professional behaviour: capacity Health professional behaviour: capacity Yes purchase group contract obligations to plan change to plan change Capacity for organizational change: – No regulations, rules and policies – Incentives and resources: financial No incentives and disincentives – Capacity for organizational change: No mandate, authority and accountability – Social, political and legal factors: No economic constraints on the health care budget – Social, political and legal factors: No contracts Lack of responsiveness to AMDEs from Health professional cognitions: expected – No representatives or manufacturers outcome – Health professional cognitions: intention No and motivation – Health professional behaviour: self- No monitoring or feedback – Social, political and legal factors: No influential people Sub-total unique or matching determinants 3 8 1/10 (10.0%) Total unique or matching determinants 10 19 9/37 (24.3%) Across both mappers, themes relating to physician be- multiple themes pertinent to physician beliefs and device liefs were mapped to 4 unique TDF domains, while orga- market (Table 1). Similarly, the TICD determinant nizations or systems and device market were each ‘Health professional cognitions: expected outcome’ was mapped to 5 unique domains. Overall, the TDF identi- applied across multiple themes pertinent to physician fied 7 unique domains across all AMDE reporting beliefs and device market (Table 2). themes. Using the TICD, physician beliefs themes were mapped to 3 unique determinants; policies, processes or Comparison across mappers systems themes were mapped to 14 unique determi- The two mappers differed in the number and domains nants; and device market themes were mapped to 10 or determinants matched to AMDE reporting themes, unique determinants. Overall, the TICD identified 21 revealing the subjectivity inherent in the mapping unique determinants across all AMDE reporting themes. process (Tables 1 and 2). For example, both applied the TDF domain ‘Environmental context and resources’ to Domains and determinants were convergent across themes the theme ‘No hospital, national or international systems Although AMDE reporting themes were identified at the for AMDE reporting’. For the same theme ARG also physician, organization or system, and device market chose the TDF domain ‘Reinforcement’ and LD also levels, selected domains or determinants were often chose the TDF domains ‘Knowledge’ and ‘Behavioural mapped to multiple themes. For example, the TDF do- regulation’. For the same theme, both mappers applied main ‘Beliefs about consequences’ was applied across the TICD determinants ‘Incentives and resources: Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 7 of 14 information system’, ‘Incentives and resources: availability Many interventions were identified of necessary resources’ and ‘Capacity for organizational Both frameworks identified numerous interventions for change: regulations, rules and policies’. LD also chose each AMDE reporting theme. For example, the theme the TICD determinant “Health professional knowledge ‘AMDEs were considered unexpected or unavoidable’ was and skills: domain knowledge” and ARG also chose the mapped by both mappers to the TDF domain of ‘Beliefs TICD determinants ‘Capacity for organizational change: about consequences’, for which 23 distinct interventions monitoring and feedback’, ‘Health professional cogni- are suggested across 4 categories (covert learning, com- tions: intention and motivation’, ‘Health professional be- parison of outcomes, natural consequences, and reward haviour: self-monitoring or feedback’, ‘Incentives and and threat). The same theme was mapped by both LD resources: non-financial incentives and disincentives’ and ARG to the TICD determinant of ‘Health profes- and ‘Incentives and resources: quality assurance and pa- sional cognitions: expected outcome’, for which 2 distinct tient safety systems’. Overall LD applied more TDF do- interventions are suggested (information or educational mains and fewer TICD determinants compared with strategies that provide compelling evidence, and audit ARG, potentially reflecting their individual familiarity and feedback). with the respective frameworks. For all 20 TDF domains Using the TDF, domains selected by both mappers iden- selected across both mappers for all themes, there were tified a total of 47 unique intervention options across all 7 (35.0%) matches across both mappers. For all 37 TICD themes; this included 23 unique interventions to address determinants selected across both mappers for all physician beliefs, 14 unique for organization or system themes, there were 9 (24.3%) matches across both map- themes, and 35 for device market themes. Using the pers. Thus the proportion of discrepancies across map- TICD, determinants selected by both mappers identified pers was relatively consistent across the application of 12 unique intervention options, including 2 unique inter- both frameworks. ventions for physician beliefs, 8 for organization or system themes, and 4 for device market themes. Comparison across theoretical frameworks Table 3 summarizes the TDF domains and TICD deter- Convergence of interventions minants chosen by one or both mappers for each AMDE As was noted previously, selected domains or determi- reporting theme. A greater number of TICD determi- nants were often similar across AMDE reporting themes nants were applied overall across themes and mappers and determinant levels. Hence, interventions recom- compared with TDF domains. This could be attributed mended by the TDF and TICD were also similar. For ex- to the level of the specificity corresponding to interven- ample, across themes describing physician beliefs, tion identification (domains for the TDF and determi- interventions frequently recommended by TDF included nants for the TICD) or the focus of the frameworks covert learning, comparison of outcomes, natural conse- themselves. The TDF largely focuses on determinants of quences, and reward and threat. Common interventions individual behaviour while the TICD offers determinants recommended by TICD included information or educa- at the individual, organization or system, and market tional strategies that provide compelling evidence or ad- levels, thus better aligning with the multi-level nature of dress reasons for disagreement, audit and feedback, and determinants contributing to AMDE reporting. How- a local consensus process. ever, several TDF domains were similar in meaning to TICD determinants, albeit identified by different labels. For example, themes relating to physician beliefs were Direct relevance of interventions mapped to the TDF domain ‘Beliefs about consequences’ In some cases, interventions recommended by the TDF and the TICD determinant ‘Health professional cogni- and TICD were intuitively linked to the determinant tions: expected outcome’ and policies, processes or theme. For example, the theme ‘Views about cause of systems themes were mapped to the TDF domain ‘Envir- AMDEs confounded by multiple factors’ was mapped to onmental context and resources’ and the TICD deter- the TDF domain ‘Knowledge’, for which 17 interventions minant ‘Incentives and resources: information system’. were recommended in the categories of feedback and Matching of TDF domains and TICD determinants was monitoring and shaping knowledge and natural conse- apparent across all themes and levels. quences, which both reflect knowledge sharing. The same theme was mapped to the TICD determinant of Interventions corresponding to TDF domains and TICD ‘Health professional knowledge and skills: domain know- determinants ledge’ for which 3 interventions were recommended, in- Additional file 2 summarizes the interventions corre- cluding change the mix of professional skills; tailor sponding to TDF domains and TICD determinants se- educational strategies; and disseminate new knowledge, lected by one or both mappers. again all focused on knowledge sharing. Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 8 of 14 Table 3 Comparison of determinant mapping across theoretical frameworks Determinant themes from AMDE study TDF domains selected TICD domains:determinants selected Apparent match in underlying meaning Both mappers One mapper Both mappers One mapper PHYSICIAN BELIEFS AMDEs considered expected or Beliefs about Social-professional role and Health professional Health professional cognitions: Yes (expected outcome) unavoidable and not adverse unless consequences identity cognitions: expected agreement with the recommendation outcomes catastrophic; viewed as outcome more severe in other specialties AMDEs within 2 years of use were Beliefs about – Health professional Health professional cognitions: Yes (expected outcome) considered unusual consequences cognitions: expected agreement with the recommendation outcome Views about cause of AMDEs – Beliefs about consequences, – Health professional cognitions: Yes (knowledge) confounded by multiple factors Knowledge agreement with the recommendation, Health professional knowledge and skills: domain knowledge Incidence of AMDEs has decreased, – Beliefs about consequences, Health professional Health professional cognitions: Yes (expected outcome) thus devices were thought to be optimism cognitions: expected agreement with the recommendation improved outcome POLICIES, PROCESSES OR SYSTEMS Follow-up of device-related outcomes Environmental context Social-professional role and Recommended behaviour: Health professional cognitions: intention Yes (professional role or beyond short-term results done and resources identity observability and motivation, Health professional behaviour, observability or elsewhere behaviour: nature of the behaviour, knowledge of own Health professional knowledge and behaviour) skills: knowledge about own practice, Health professional behaviour: self- monitoring or feedback, Professional interactions: referral processes Devices implanted not recorded in Environmental context – Incentives and resources: Health professional knowledge and Yes (resources or information patient records and resources information system skills: knowledge about own practice, system) Health professional behaviour: self- monitoring or feedback, Health professional behaviour: capacity to plan change No hospital, national or international Environmental context Knowledge, Reinforcement, Incentives and resources: Health professional knowledge and Yes (resources or information systems for AMDE reporting and resources Behavioural regulation information system, skills: domain knowledge, Health system, knowledge, Incentives and resources: professional cognitions: intention and reinforcement or non- availability of necessary motivation, Health professional financial incentives or resources, Capacity for behaviour: self-monitoring or feedback, disincentives, regulation or organizational change: Incentives and resources: non-financial self- or organizational regulations, rules and incentives and disincentives, Incentives monitoring) policies and resources: quality assurance and patient safety systems, Capacity for organizational change: monitoring and feedback Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 9 of 14 Table 3 Comparison of determinant mapping across theoretical frameworks (Continued) Determinant themes from AMDE study TDF domains selected TICD domains:determinants selected Apparent match in underlying meaning Both mappers One mapper Both mappers One mapper DEVICE MARKET Use of specific devices often Environmental context – Health professional Capacity for organizational change: Yes (context and resources determined by purchase group and resources behaviour: capacity to plan regulations, rules and policies, or policies, financial contract obligations change Incentives and resources: financial incentives and disincentive, incentives and disincentives, Capacity authority, budget, contracts) for organizational change: mandate, authority and accountability, Social, political and legal factors: economic constraints on the health care budget, Social, political and legal factors: contracts Lack of responsiveness to AMDEs Reinforcement Knowledge, Optimism, – Health professional cognitions: expected Yes (reinforcement or from industry Beliefs about consequences, outcome, Health professional cognitions: feedback or influential Environmental context and intention and motivation, Health people, expected outcome, resources professional behaviour: self-monitoring or optimism or motivation) feedback, Social, political and legal factors: influential people Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 10 of 14 In other cases, the applicability of interventions rec- TDF interventions (categorized as scheduled conse- ommended by the TDF and TICD appeared less direct, quences) focused on adding or removing rewards, while perhaps owing to a greater degree of complexity in de- the TICD interventions (information or educational terminants identified in the primary study. For example, strategies and audit and feedback) focused on the at the device market level, the theme ‘Use of specific provision of information. devices often determined by purchasing group contract obligations’ was mapped to the TDF domain of ‘Environ- Implications for practice mental context and resources’ for which 14 interventions Table 4 summarizes overall study findings and their im- categorized as antecedents or associations were recom- plications. Knowledge generated by this study addresses mended. These interventions involve restructuring the the applied objectives of identifying interventions to physical or social environment, or adding or removing stimulate AMDE reporting, and comparing the domains prompts or cues, and do not seem to readily address the or determinants and interventions identified by mapping multi-level restrictions on behaviour of purchasing AMDE reporting themes to the TDF and TICD. group contracts. Conversely, mapping the same theme to TICD determinants identified the more granular Interventions to stimulate AMDE reporting intervention of improvements in contracts. AMDE reporting themes were mapped by both mappers Similarly, important themes from the predicate study to several domains and determinants, which identified reflecting complex determinants may not have been corresponding interventions common to the TDF and well-addressed by either TDF or TICD, leading to less TICD. Information or educational strategies that provide than appropriate interventions. For example, the theme evidence about AMDEs, and audit and feedback of ‘Views about cause of AMDEs confounded by multiple AMDE-related data were identified as interventions to factors’ was mapped to the TDF domain of ‘Beliefs about target physician beliefs; improve information systems, consequences’ by both mappers and the TICD domain and reminder cues, prompts and awards were identified of ‘Health professional cognitions: expected outcome’ by to target organization or system themes; and modify fi- both mappers, ultimately leading to 23 corresponding nancial/non-financial incentives, and share data on out- interventions recommended by TDF and 5 recom- comes associated with AMDEs were identified to mended by TICD. All of the interventions address address device market themes. However, issues and dis- knowledge but none appear to fully recognize the inter- crepancies in the application of TDF and TICD raise un- play of determinants inherent in this theme. certainty about which or how many interventions may be relevant to promote and support AMDE reporting. Comparison across theoretical frameworks Overall, although a greater number of TICD determi- Application of the TDF and TICD nants were applied across themes and mappers com- Issues revealed by this study include a lack of clarity pared with TDF domains, the TDF identified many more about how directly relevant domains or determinants unique interventions across all themes (47 for domains should be and therefore which and how many to select; selected by both mappers plus additional domains se- if and how to resolve discrepancies in the selection of lected by one mapper) compared with the TICD (12 in- domainsordeterminantsacrossmultiplemappers;and terventions for determinants selected by both mappers how to choose interventions from among the large plus additional determinants selected by one mapper). number associated with selected domains and determi- Several interventions recommended by TDF and TICD nants. Several TDF domains and TICD determinants were similar in meaning, irrespective of the theme. For were relevant, similar in meaning, and selected by both example, for the physician beliefs theme ‘AMDEs consid- mappers. Convergence within and across TDF and ered expected or unavoidable and not adverse’, the TDF TICD identified a core set of behavioural determinants intervention of comparison of outcomes was conceptu- and corresponding interventions. Thus, both theoretical ally similar to the TICD intervention of audit and feed- frameworks were useful for selecting behavioural deter- back, and the TDF intervention of information about minants to which AMDE reporting themes matched health consequences was similar to the TICD interven- and corresponding interventions. tion of information or educational strategies that provide However, TDF domains and TICD determinants se- compelling evidence. lected independently by both mappers often did not Even when themes were mapped to domains or deter- match, and a large number of interventions corre- minants that were similar in meaning, different interven- sponded to the TDF domains and TICD determinants tions were recommended by TDF and TICD in some selected by one or both mappers. Even when themes instances. For example, for the device market theme mapped to TDF domains and TICD determinants with ‘Lack of responsiveness to AMDEs from industry’, the similar definitions, the frameworks often recommended Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 11 of 14 Table 4 Summary of findings and implications Finding Implication All AMDE reporting themes mapped to both TDF and TICD Both theoretical frameworks were useful for systematically analyzing AMDE reporting determinant themes Multiple TDF domains and TICD determinants were relevant Provide users with flexibility to choose and further prioritize from among the array of relevant domains/determinants but also raises uncertainty about how many to choose and with what precision Several TDF domains and TICD determinants chosen by one or both Convergence across frameworks could be used to identify a core set of mappers were conceptually similar though labelled differently behavioural determinants Selected TDF domains and TICD determinants chosen by one or both Convergence within frameworks could be used to identify a core set of mappers applied to more than one AMDE reporting theme behavioural determinants Domains and determinants selected independently by two mappers Selection of TDF domains and TICD determinants may be subjective and often did not match; discrepancy rate similar for TDF and TICD influenced by mapper familiarity with a given theoretical framework. It is unclear if a process is needed to resolve discrepancies or, instead, if intervention design should be based on only domains/determinants selected by all independent mappers, or on a core set of domains/ determinants most commonly selected by all mappers Greater number of TICD determinants were applied across themes and Compared with TDF, which focuses on individual level domains, TICD mappers compared with TDF domains offers multilevel determinants, plus definitions and examples for each, and was thus easier to apply and could be applied with greater precision at a granular level Numerous interventions corresponded to common TDF domains and It is unclear how to choose the intervention (s) that are most relevant TICD determinants selected by both mappers for each AMDE reporting from among the large number of options presented by the TDF and theme TICD Additional interventions corresponded to TDF domains and TICD It is unclear if intervention (s) should be chosen based on only those determinants selected by one mapper associated with domains/determinants selected by all independent mappers, or with a core set of domains and determinants most commonly selected by all mappers Given that similar TDF domains and TICD determinants were applied Convergence within frameworks could be used to identify a core set of across AMDE reporting themes, corresponding interventions were also interventions corresponding to behavioural determinants convergent Some interventions recommended by TDF and TICD for the same AMDE Convergence across frameworks could be used to identify a core set of reporting themes were conceptually similar though labelled differently interventions corresponding to behavioural determinants Although more TICD determinants were applied compared with TDF It is unclear if and how interventions that are most relevant for a given domains, TDF recommended a greater number of interventions context should be screened or prioritized from among the options compared with TICD recommended by either TDF or TICD Even when themes mapped to conceptually similar TDF domains and It is unclear how to choose the intervention (s) that are most relevant TICD determinants, TDF and TICD often recommended conceptually when two rigorously developed theoretical frameworks differ in the different interventions interventions recommended for the same determinant Some interventions recommended by TICD seemed more intuitively Compared with TDF, which recommends interventions corresponding to relevant compared with TDF broad domains, TICD recommends interventions corresponding to specific determinants, and may identify interventions that are more relevant. Following the mapping of themes to theoretical frameworks, consultation with stakeholders is likely needed to deliberate the relevance and feasibility of corresponding interventions for a given context. Complex determinants involving interplay among factors were not well- Domains and corresponding interventions in the TDF or TICD did not addressed by TDF or TICD fully recognize the complex interplay of determinants inherent in some themes. It is unclear if this is because the frameworks are better suited to exploring determinants in some contexts (i.e. adherence with clinical guideline recommendations) and not others (i.e. reporting of AMDEs. Neither TDF nor TICD prompt users to prioritize domains or interventions Neither the TDF nor the TICD prompt users to prioritize among the many potentially applicable domains or interventions as means of limiting or focusing the number and type of interventions different interventions. TICD recommended interventions the complex interplay of determinants inherent in some that seemed to be more directly applicable to a behavior themes; it is unclear if this is because the frameworks are such as AMDE reporting with multi-level determinants as better suited to exploring determinants in some contexts compared with the TDF. Domains and corresponding in- (i.e. adherence with clinical guideline recommendations) terventions in the TDF or TICD did not fully recognize and not others (i.e. reporting of AMDEs. Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 12 of 14 Discrepancies in applying TDF and TICD may be would maximize change”, underscoring the need for accounted for by distinctions between their content and “broader contextual consideration”. One potential ex- format. TDF includes determinant domains largely fo- planation is reality that theoretical frameworks do not cused on the individual level while TICD includes deter- address causal mechanisms, or how change occurs, minant domains and determinants spanning multiple which presents a challenge when attempting to identify levels and, unlike TDF, offered definitions and examples which intervention (s) are most likely to support im- to guide the application of these more granular determi- provement . Phillips et al. interviewed 10 health care nants. Although more TICD determinants were applied professionals from six disciplines who used the TDF compared with TDF domains, TDF recommended a . Frequently cited challenges experienced when ap- greater number of interventions compared with TICD. plying it included the time and resources required to use While the predicate study did not itself prioritize deter- the TDF, lack of clear operational definitions, and over- minants, neither the TDF nor the TICD prompt users to lap between domains. Participants found it difficult, prioritize among the many potentially applicable do- complicated, unwieldy, and subjective to interpret and mains or interventions as means of limiting or focusing apply the domains . Birken et al. conducted a sys- the number and type of interventions. Overall, uncer- tematic review of five protocols and seven studies that tainty remains about the optimal way to identify inter- used both the TDF and the Consolidated Framework for ventions that match behavioural determinants for a Implementation Research (CFIR) to examine the ration- given behaviour, and the precision and relevance of ale for having applied both frameworks . Authors of those choices. included studies justified the use of both frameworks by stating that one offered greater insight on determinants Discussion and the other on interventions, although which frame- This study was a naturalistic application of the TDF and work offered determinants versus interventions was TICD to identify evidence-based interventions corre- interchangeable across studies. A conceptual analysis of sponding with known determinants of AMDE reporting reasons for the failure of interventions designed based and, in so doing, to explore how use of these theoretical on the TICD offered several reasons including potential frameworks could be optimized. Both TDF and TICD mismatch of determinants to interventions or a subse- were useful in identifying several interventions that quent mismatch of interventions to targeted groups and could promote and support AMDE reporting. However, settings . Thus, our research and that of others reveals it is uncertain which interventions are the best options uncertainty and challenges in the application of theoret- given discrepancies in the selection of TDF domains and ical frameworks to design behaviour change interven- TICD determinants, and corresponding interventions tions. More recently a guide to use of the TDF was across theoretical frameworks and independent mappers. published . The guide specifies that coding disagree- The content and format of TICD (well-defined domains ments could be resolved by either consensus among and determinants spanning individual, organizational, coders or assessment of inter-rater reliability, and when system and environmental levels) may make it easier to uncertain about coding to apply all relevant TDF do- apply than the TDF for individuals who are not familiar mains. However, these suggestions do not help users se- with either framework. Even still, uncertainty remains lect from among the many potential interventions about how to best apply the frameworks in practice and identified by this approach. their precision when used to design behaviour change The interpretation and application of these findings interventions. may be limited by several factors. Independent mappers Our findings align with previous work highlighting the made a deliberate decision to not coordinate their inter- uncertainty and challenges surrounding the application pretation of the TDF and TICD before mapping, nor did of theoretical frameworks to design behaviour change in- they intend to discuss and resolve discrepancies after terventions. Lipworth et al. analyzed determinants of the mapping. The objective was to independently apply the uptake of clinical quality interventions and found that all theoretical frameworks specifically to explore the nature 14 TDF domains and numerous corresponding interven- of any arising discrepancies as a means of identifying tions were relevant, necessitating a “drilling down” to problems that may be encountered by others when identify those that were most “contextually salient” . employing these tools in implementation planning. Each Lawton et al. used the TDF to conduct and analyze the mapper had differing levels of familiarity with both findings of 60 interviews with 60 general practice health frameworks, thereby precluding the ability to comment care professionals regarding adherence to various clinical on the nature of discrepancies when those applying the recommendations . A wide variety of determinants framework have similar levels of experience. As ARG were identified but it was difficult to “pinpoint which de- conducted the interviews for the primary study, it is pos- terminants, if targeted by an implementation strategy, sible her familiarity with the data may have led to a Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 13 of 14 contextual advantage when applying the frameworks. which or how many interventions may be relevant to pro- The challenges and discrepancies encountered when ap- mote and support AMDE reporting. Given the worldwide plying the frameworks may be specific to the single case imperative to promote the use of evidence-based innova- examined, that of determinants of AMDE reporting. tions and improve the quality and safety of care, there is Also, the TDF and TICD may be better suited to asses- an urgent need to make tools such as the TDF and TICD sing determinants and corresponding interventions for easier to use for a broader audience, and to establish a some contexts more so than others; that could not be reliable way to identify which of many potential interven- determined by this study and will require future tions are likely to successfully address specific determi- research. nants. Just as research more broadly has seen a shift from With respect to selecting determinants and interven- the production and dissemination of evidence to the im- tions, our research and that of others [3, 6, 25–27]found plementation of evidence, and it is time for the field of im- that the TDF and TICD are useful for fully describing the plementation science to shift from the development of range of potentially relevant determinants, a task perhaps frameworks to supporting their application in practice. best done by implementation scientists who are familiar with the constructs and their definitions. This suggests Additional files that selecting the most relevant determinants and inter- ventions is likely to benefit from collaboration with stake- Additional file 1: AMDE reporting themes that emerged from previous holders with context-specific knowledge. Processes such study  (DOCX 17 kb). as Intervention Mapping, whereby researchers and health Additional file 2: Comparison of interventions identified across theoretical frameworks (DOCX 30 kb). care professionals can jointly choose and design interven- tions based on the identification and prioritization of determinants, may prove useful for developing and evalu- Abbreviations ating interventions that are more likely to improve the de- AMDE: Adverse medical device event; ARG: Anna R Gagliardi; LD: Laura Desveaux; TDF: Theoretical Domains Framework; TICD: Tailored livery and outcomes of care . Implementation for Chronic Diseases Further research applying the TDF and TCID in spe- cific contexts is needed in order to resolve the differ- Funding ences between them and clarify the circumstances for This study was funded by the Canadian Institutes for Health Research, who which each framework is most useful. The critical need took no part in the design of the study; data collection, analysis or remains to make these tools easier to use for a broader interpretation; or in the writing of the manuscript. audience, and to establish a reliable way to identify which of many potential interventions are likely to suc- Availability of data and materials All data generated or analysed during this study are included in this cessfully address specific determinants. Key consider- published article and its supplementary information files. ations include how many independent mappers are needed, what process is needed to resolve discrepancies Authors’ contributions across mappers, whether intervention design should be ARG conceptualized the study and acquired funding; designed the study, based on only those domains or determinants selected collected and analysed data, drafted the manuscript, and gave final approval of the version to be published. LD assisted with study design, collected and by all independent mappers, or on some other combin- analyzed data, drafted the manuscript and gave final approval of the version ation of domains or determinants identified; and how to be published. best to prioritize the selection of potential interventions. Further insight or framework development is also Ethics approval and consent to participate needed to help users address complex determinants, and This study, based on secondary analysis of themes that emerged from a previous study, did not require ethical approval. For the previous study , to prioritize domains and corresponding interventions. ethical approval was granted by the University Health Network Research Ethics Board and all participants provided written informed consent prior to Conclusions being interviewed. The TDF and TICD were employed to identify behav- ioural interventions corresponding to determinants of Competing interests The authors declare that they have no competing interests. the reporting of AMDEs. Interventions common to both frameworks included information or educational strat- egies that provide evidence about AMDEs; audit and Publisher’sNote feedback of AMDE data; improved information systems; Springer Nature remains neutral with regard to jurisdictional claims in reminder cues, prompts and awards; modifying finan- published maps and institutional affiliations. cial/non-financial incentives; sharing data on outcomes Author details associated with AMDEs. Challenges and discrepancies in 1 2 Women’s College Hospital, Toronto, Canada. University Health Network, the application of frameworks raise uncertainty about Toronto, Canada. Desveaux and Gagliardi BMC Health Services Research (2018) 18:402 Page 14 of 14 Received: 1 August 2017 Accepted: 29 May 2018 21. Gagliardi AR, Brouwers MC. Do guidelines offer implementation advice to target users?: a systematic review of guideline applicability. BMJ Open. 2015; 5:e007047. 22. Gagliardi AR, Dobrow MJ. Identifying the conditions needed for integrated References knowledge translation (IKT) in health care organizations: qualitative 1. Nilsen P. Making sense of implementation theories, models and frameworks. interviews with researchers and research users. BMC Health Serv Res. 2016; Implement Sci. 2015;10:53. 16:256. 2. Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. 23. Willson M, Vernooij RW, Gagliardi AR. Members of the guidelines Tailored interventions to address determinants of practice. Cochrane international network implementation working group. Questionnaires used Database Syst Rev. 2015;4:CD005470. to assess barriers of clinical guideline use among physicians are not 3. Michie S, Wood CE, Johnston M, Abraham C, Francis JJ, Hardeman W. comprehensive, reliable or valid: a scoping review. J Clin Epidemiol. 2017;86: Behaviour change techniques: the development and evaluation of a 25–38. taxonomic method for reporting and describing behaviour change 24. Liang L, Bernhardsson S, Vernooij RWM, Armstrong M, Bussieres A, Brouwers interventions (a suite of five studies involving consensus methods, MC, et al. Use of theory to plan or evaluate guideline implementation randomised controlled trials and analysis of qualitative data). Health Technol among physicians: a scoping review. Implement Sci. 2017;12:26. Assess. 2015;19:1–188. 25. Lipworth W, Taylor N, Braithwaite J. Can the theoretical domains framework 4. Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, account for the implementation of clinical quality interventions? Health Serv et al. A checklist for identifying determinants of practice: a systematic review Res. 2013;13:530. and synthesis of frameworks and taxonomies of factors that prevent or enable 26. Phillips CJ, Marshal AP, Chaves NJ, Jankelowitz SK, Lin EB, Loy T, et al. improvements in healthcare professional practice. Implement Sci. 2013;8:35. Experiences of using the theoretical domains framework across diverse 5. Phillips CJ, Marshall AP, Chaves NJ, Jankelowitz SK, Lin IB, Loy CT, et al. clinical environments: a qualitative study. J Multidisc Healthc. 2015;8:139–46. Experiences using the theoretical domains framework across diverse clinical 27. Birken SA, Powell BJ, Presseau J, Kirk MA, Lorencatto F, Gould NJ, et al. environments: a qualitative study. J Multidiscip Healthc. 2015;8:139–46. Combined use of the consolidated framework for implementation research 6. Wensing M. The tailored implementation in chronic diseases (TICD) project: (CFIR) and the theoretical domains framework (TDF): a systematic review. introduction and main findings. Implement Sci. 2017;12:5. Implement Sci. 2017;12:2. 7. Murphy K, O’Connor DA, Browning CJ, French SD, Michie S, Francis JJ, et al. 28. Atkins L, Francis J, Islam R, O’Connor D, Patey A, Ivers N, et al. A guide to Understanding diagnosis and management of dementia and guideline using the theoretical domains framework of behaviour change to implementation in general practice: a qualitative study using the theoretical investigate implementation problems. Implement Sci. 2017;12:77. domains framework. Implement Sci. 2014;9:31. 29. Bartholomew LK, Parcel GS, Kok G. Intervention mapping: a process for 8. Lawton R, Heyhoe J, Louch G, Ingleson E, Glidewell L, Willis TA, et al. Using developing theory- and evidence-based health education programs. Health the theoretical domains framework (TDF) to understand adherence to Educ Behav. 1998;25:545–63. multiple evidence-based indicators in primary care: a qualitative study. Implement Sci. 2016;11:113. 9. Maisel WH. Medical device regulation: an introduction for the practicing physician. Ann Intern Med. 2004;140:296–302. 10. Maisel WH, Moynahan M, Zuckerman BD, Gross TP, Tovar OH, Tillman DB, et al. Pacemaker and ICD generator malfunction. JAMA. 2006;295:1901–6. 11. Carr AJ, Robertsson O, Graves S, Price AJ, Arden NK, Judge A, et al. Knee replacement. Lancet. 2012;379:1331–40. 12. Lawton R, McEachan RRC, Giles SJ, Sirriyeh R, Watt IS, Wright J. Development of an evidence-based framework of factors contributing to patient safety incidents in hospital settings: a systematic review. BMJ Qual Saf. 2012;21:369–80. 13. Shuren J, Califf RM. Need for a national evaluation system for health technology. JAMA. 2016;316:1153–4. 14. Gagliardi AR, Ducey A, Lehoux P, Turgeon T, Ross S, Trbovich P, et al. Determinants of reporting adverse medical device events: qualitative interviews with physicians about higher-risk implantable devices. BMJ Qual Saf. 2018;27:190-8. 15. Ivers NM, Desveaux L, Presseau J, Reis C, Witteman HO, Taljaard MK, et al. Testing feedback message framing and comparators to address prescribing of high-risk medications in nursing homes: protocol for a pragmatic, factorial, cluster-randomized trial. Implement Sci. 2017;12:86. 16. Desveaux L, Gomes T, Tadrous M, Jeffs L, Taljaard M, Rogers J, et al. Appropriate prescribing in nursing homes demonstration project (APDP) study protocol: pragmatic, cluster-randomized trial and mixed-methods process evaluation of an Ontario policy-makers initiative to improve appropriate prescribing of antipsychotics. Implement Sci. 2016;11:1–10. 17. Ivers NM, Schwalm JD, Witteman HO, Presseau J, Taljaard M, McCready T, et al. Interventions supporting long-term adherence aNd decreasing cardiovascular events (ISLAND): pragmatic randomized trial protocol. Am Heart J. 2017;190:64–75. 18. Honein G, Stuart-McEwan T, Waddell T, Salvarrey A, Smylie J, Dobrow M, et al. How do organizational characteristics influence teamwork and service delivery in lung cancer diagnostic assessment programs? A mixed methods study. BMJ Open. 2017;7:e013965. 19. Gagliardi AR, Nathens AB. Exploring the characteristics of high-performing hospitals that influence trauma triage and transfer. J Trauma Acute Care Surg. 2015;78:300–5. 20. Gagliardi AR, Straus SE, Shojania KG, Urbach DR. Multiple interacting factors influence adherence and outcomes associated with surgical safety checklists: a qualitative study. PLoS One. 2014;9:e108585.
BMC Health Services Research – Springer Journals
Published: Jun 4, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera