Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Interest in a Mobile App for Two-Way Risk Communication: A Survey Study Among European Healthcare Professionals and Patients

Interest in a Mobile App for Two-Way Risk Communication: A Survey Study Among European Healthcare... Drug Saf (2018) 41:697–712 https://doi.org/10.1007/s40264-018-0648-0 OR IGINAL RESEARCH ARTIC L E Interest in a Mobile App for Two-Way Risk Communication: A Survey Study Among European Healthcare Professionals and Patients 1 1 2 2 • • • • Sieta T. de Vries Petra Denig Carmen Lasheras Ruiz Franc ¸ois Houy ¨ ez 3 3 1 • • • Lisa Wong Alastair Sutcliffe Peter G. M. Mol on behalf of IMI Web-RADR Work Package 3b Consortium Published online: 2 March 2018 The Author(s) 2018. This article is an open access publication Abstract characteristics on the level of interest in the app (i.e. very Introduction Previously, an app has been developed for interested vs. not/somewhat interested). healthcare professionals (HCPs) and patients to report Results In total, 399 HCPs and 656 patients completed the adverse drug reactions (ADRs) to national medicines survey. About half of the patients (48%; ranging from 38% agencies and to receive drug safety information. from The Netherlands to 54% from The UK), and 61% of the Objective This study aimed to assess (1) European HCPs’ and HCPs (ranging from 42% from The Netherlands to 54% from patients’ interest in an app for this two-way risk communica- The UK) were very interested in the app. A faster means of tion; (2) their preferences and perceptions towards specific app reporting ADRs and easier access to the reporting form were characteristics; and (3) which HCPs and patients are particu- the main perceived benefits. HCPs and patients who already larly interested in the app. In addition, these aspects were use a health app were particularly interested in the app studied specifically for the countries where such an app was (HCPs: odds ratio [OR] 3.52; 95% confidence interval [CI] already available, i.e. Croatia, The Netherlands, and The UK. 1.96–6.30, patients: OR 1.64; 95% CI 1.19–2.27). Methods European HCPs and patients were asked to Conclusions An app is positively perceived by HCPs and complete a web-based survey developed in the context of patients for reporting ADRs quickly and for receiving drug the Web-Recognizing Adverse Drug Reactions (Web- safety information from national medicines agencies. In RADR) project. Data on app interest and preferences and particular, HCPs and patients who already use other health perceptions towards app characteristics were analysed apps were interested in the app. descriptively. Logistic regression analyses were conducted to assess the association of HCP characteristics and patient Key Points The members of IMI Web-RADR Work Package 3b Consortium are Interest in an app for two-way risk communication listed in acknowledgements. (i.e. to report adverse drug reactions [ADRs] to national medicines agencies and to receive drug Electronic supplementary material The online version of this article (https://doi.org/10.1007/s40264-018-0648-0) contains supple- safety information) is high among healthcare mentary material, which is available to authorized users. professionals (HCPs) and patients. & Peter G. M. Mol The app should be a faster way to report ADRs than [email protected] conventional reporting options and should preferably offer additional information about drug–drug Department of Clinical Pharmacy and Pharmacology, interactions and previously reported ADRs. University of Groningen, University Medical Center Groningen, Groningen, The Netherlands Strategies to disseminate an app on two-way risk European Organisation for Rare Diseases (Eurordis), Paris, communication could focus on targeting HCPs and France patients who already use a health app since these Population, Policy and Practice Programme, UCL Institute of persons were particularly interested in the app. Child Health, 30 Guilford Street, London WC1N 1EH, UK 698 S. T. Vries et al. The aim of the current study was to assess (1) European 1 Introduction HCPs’ and patients’ interest in an app for two-way risk communication; (2) their preferences and perceptions Healthcare professionals (HCPs) and patients have access towards specific characteristics of the app; and (3) which to a plethora of health-related mobile apps but not every HCPs and patients are particularly interested in such an person is equally interested in such apps. Recently, a health app. In addition, these aspects were specifically studied for app developed in the context of the Web-Recognizing the countries where such an app was already available, i.e. Adverse Drug Reactions (Web-RADR) project (https:// Croatia, The Netherlands and The UK. This knowledge can web-radr.eu/) was added to the available health-related be used by national medicines agencies in the development apps. The goal of this app is to provide two-way risk or improvement of an app for two-way risk communication communication, defined as the possibility to report adverse and in the development of strategies to inform potential drug reactions (ADRs) to national medicines agencies/ users about the existence of the app. pharmacovigilance centres, and to receive drug safety information from these agencies [1]. Previously, in a qualitative study, we identified various factors that may 2 Methods influence the use of this app and showed that HCPs and patients were generally positive about its development [2]. 2.1 Study Design and Survey Development However, these aspects should be validated in a larger population. In this cross-sectional study, data were collected between Considering the plethora of new technologies, including July and October 2016 using web-based surveys. Two apps, theoretical models have been developed attempting surveys (i.e. one for HCPs and one for patients) were to identify factors that influence the uptake of the new developed in English by members of the Web-RADR technology. An example is the Unified Theory of Accep- project (see Electronic Supplementary Material 1 and 2 for tance and Use of Technology, which states that user the HCP and patient survey, respectively). The English- characteristics play a moderating role in the acceptance of language surveys were translated by an official translation technology [3]. This indicates that not every HCP or agency into Croatian, Dutch, French, German, Portuguese patient will be interested in the app. Several studies have and Spanish. Web-RADR members checked whether the investigated whether interest in health apps is influenced by translations had the same meaning as the English version. characteristics such as age, ethnicity/race, gender, current The web-based format of the surveys was created using use of a health app, inability to work, income, educational Unipark software (http://unipark.com/en/). A separate link degree, clinical characteristics or having a family member was available for the HCP and patient survey in each of the with a specific disease [4–8]. Although the studies con- languages. sistently show that older people are generally less inter- The content of the surveys was based on the results of a ested in health apps than younger people [4–8], the qualitative study [2], input from members of other work literature is inconclusive about other user characteristics. packages of the Web-RADR project, and various HCP and For instance, Latinos/Hispanics were less interested in one patient organisations. The patient survey contained ques- study [6] but more interested in another study [8] than tions about ADR reporting in general; their opinion of an Caucasians/white people. Likewise, one study showed that app to report ADRs, an app to receive safety information, males were slightly less interested than females [7], and an app for two-way risk communication; reporting whereas another study showed no gender differences [8]. ADRs through an app of the national medicines agency; The previous studies conducted in the USA [5–8]or safety information and receiving such information through Asia (i.e. Singapore) [4] focused on patients or the general an app; and, finally, some general questions such as age, population, and assessed a person’s interest in a health app gender and the country in which they lived at the time of in general [6–8] or in apps to support adherence or self- survey completion. management [4, 5]. A recent study about the VigiBIP app, In some questions, the name of the national medicines developed by the Toulouse University Pharmacovigilance agency/pharmacovigilance centre was mentioned. The Center for two-way risk communication, suggests that Agenciju za lijekove i medicinske proizvode (HALMED), patients are interested in the app and that different ADRs Nederlands Bijwerkingencentrum (Lareb), Medicines and may by reported via the app compared with conventional Health Regulatory Agency (MHRA), l’Agence nationale methods [9]. However, more studies on characteristics of de se ´curite ´ du me ´dicament et des produits de sante ´ HCPs and patients on their interest in apps for communi- (ANSM), Bundesinstitutes fur Arzneimittel und Medizin- cating health-related issues with national medicines agen- produkte, Autoridade Nacional do Medicamento e Produ- cies are needed. tos de Sau ´ de (INFARMED) and la Agencia Espan ˜ ola del Interest in an App for Two-Way Risk Communication 699 Medicamento y Productos Sanitarios (AEMPS) were 2.5 Determinants: Healthcare Professional (HCP)/ mentioned in the Croatian, Dutch, English, French, Ger- Patient Characteristics man, Portuguese and Spanish versions of the survey, respectively. The following HCP characteristics were assessed as determinants for HCPs’ interest in an app for two-way risk Participants gave their implied consent to participate in the study by voluntarily completing the survey. communication: age, gender, how often they already used health apps and whether they had ever reported an ADR to 2.2 Participants and Data Collection the national medicines agency. For patients, the following characteristics were assessed: age, gender, educational Any HCP or patient in Europe familiar with mobile apps level, number of medicines, how often they already used was eligible to participate in this study. All HCPs were health apps, whether they had ever experienced an ADR considered to be familiar with apps. Patients were informed and whether they were aware they could report ADRs to that they should only complete the survey if they were the national medicines agency (Table 1). familiar with apps. Various channels were used to reach HCPs and patients. For instance, European and national 2.6 Analyses HCP and patient organisations distributed the survey among members via direct e-mail or advertisements on Descriptive statistics are presented for HCPs and patients separately. In addition, this is presented for countries in their websites and/or in their newsletters. The survey was also announced on Facebook and Twitter accounts, for which the app was already available. Completers of the instance on the account of the Web-RADR project. Croatian, Dutch or English version of the survey who Recruitment strategies focused particularly on reaching indicated they were living in these countries at the time of HCPs and patients in Croatia, The Netherlands and The UK the survey were included in these country-specific analy- since the Web-RADR app on two-way risk communication ses. Differences across these countries were tested using 2 2 was available in these countries at the time of this study. Chi-squared (v ) tests. Three post hoc v -tests were con- The pharmacovigilance centres in Croatia, The Netherlands ducted in the case of P\0.05 to test which countries dif- and The UK also distributed the survey, for instance by fered from each other. The Bonferroni correction was applied for these post hoc analyses to correct for multiple posting a message on their respective websites. To encourage response rates, survey completers had the option testing. This implies that P values\0.016 were considered statistically significant. to participate in a prize draw to win a €50 coupon. Logistic regression analyses were conducted to assess associations between responder characteristics and the 2.3 Outcome Measure: Interest in the App dichotomised outcome measure, expressing a high interest The outcome measure of this study was responders’ interest in the app. For this, being very interested was contrasted in an app for two-way risk communication. Responders with being somewhat or not interested (Table 1). In sen- were asked to rate on a 4-point Likert scale to what extent sitivity analyses using generalised ordered logit models they were interested in such an app (Table 1). Responders [10, 11], we assessed whether this dichotomisation resulted could also indicate that they did not know whether they had in a loss of information. Responders were excluded from the logistic regression analyses and generalised ordered interest in the app. logit models when they (1) selected another answer option than male/female on the question about their gender; (2) 2.4 Determinants: App Characteristics did not answer or answered ‘don’t know’ on the question Expectations and actual characteristics of an app may about the app interest (outcome variable); or (3) did not influence someone’s intention to download and use an app answer a question that was used as a determinant in these [3]. Therefore, responders were asked about their prefer- analyses. ences and perceptions regarding an app for two-way risk All analyses were conducted using Stata version 13 communication. For this, questions were asked about per- (Stata Corp., College Station, TX, USA). Microsoft Excel ceived benefits in using the app, the type of news of 2010 (Microsoft Corp., Redmond, WA, USA) was used for interest, interest in other functions in the app and the the graphical presentation of the results. protection of the app. In addition, responders were asked about their intention to download an app for two-way risk communication. 700 S. T. Vries et al. Table 1 Questions and answer options used as outcome variable and determinants Variable Question Answer options Type of variable in analyses Analyses of HCPs/patients Outcome variable App interest In general, how interested would you be in an app of Not interested at all Dichotomous: not/(somewhat) HCPs and the\national medicines agency[ that you can use interested vs. very interested patients Somewhat for both, reporting side effects/adverse drug interested Don’t know ? excluded reactions and receiving safety information? Interested Very interested Don’t know Determinants Age What is your age? Continuous Continuous HCPs and patients Gender What is your gender? Male Dichotomous: male vs. female HCPs and patients Female Other/prefer not to say ? excluded Other/prefer not to say Educational What is your highest level of education completed? No formal education Dichotomous: low/secondary Patients level or below education vs. tertiary education (first and second Primary education stage) Lower secondary education Upper secondary education Post-secondary but non-tertiary education First stage of tertiary education Second stage of tertiary education Number of How many different medicines are prescribed to you 0 Categorical: 0 medicines; 1–4 Patients medicines at the moment? medicines (reference category); C 5 medicines 5 or more Use health How often do you use a health app? Daily Dichotomous: never vs. other HCPs and apps answer options patients Weekly Monthly or less often Never Experience of Have you ever experienced a side effect of a Yes Dichotomous: no/don’t know Patients ADRs medicine that you take or have taken in the past? vs. yes No Don’t know/don’t remember Awareness of Are you aware that you can report experienced side Yes Dichotomous: no/I have never Patients reporting effects to the \national medicines agency[? heard of the \national No ADRs medicines agency[ vs. yes I have never heard of the \national medicines agency[ Interest in an App for Two-Way Risk Communication 701 Table 1 continued Variable Question Answer options Type of variable in analyses Analyses of HCPs/patients Report ADR Have you ever reported an adverse drug reaction Yes Dichotomous: no/don’t know HCPs to national experienced by your patients to the \national vs. yes No medicines medicines agency[? Don’t know/don’t agency remember ADR adverse drug reaction, HCPs healthcare professionals patients. About half of the patients (48%) were very 3 Results interested in the app, which ranged from 38% in The 3.1 Characteristics of the Responders Netherlands to 54% in The UK (Fig. 1b). Interest in an app for two-way risk communication was somewhat higher 3.1.1 HCPs than interest in an app with single functionality (i.e. reporting of ADRs or receiving safety information). In total, 399 HCPs completed the survey: 192 were from 3.3 Determinants: App Characteristics Croatia, 62 were from The Netherlands, 83 were from The UK and 62 (16%) were from other European countries (i.e. 3.3.1 Perceived Benefits in Using the App countries where the app was not rolled out) (see Electronic Supplementary Material 3). The age of the responders ranged from 20 to 71 years and most of the responders With respect to the reporting functionality of the app, most of the HCPs and patients indicated that a faster way to were women (68%). Sixteen percent of the 399 HCPs indicated they had never used a health app. More than half report ADRs and easier access to the ADR reporting form were potential benefits of using the app. These answer of the HCPs had at least heard about the Web-RADR app. options were selected by 83 and 73% of the HCPs, respectively (Table 2) and by 85 and 72% of patients, 3.1.2 Patients respectively (Table 3). Keeping up-to-date with the latest drug safety news There were 656 patients who completed the survey, of (84%) and increasing their drug safety knowledge (76%) whom 136 were from Croatia, 187 were from The were important benefits perceived by HCPs on using an Netherlands, and 100 were from The UK. The remaining app. The possibility to select medicines of interest was seen 233 (36%) patients were from other European countries as the least beneficial option for HCPs (47%) (Table 2). (i.e. countries where the app was not rolled out) (Electronic Most of the patients saw it as a benefit that the app would Supplementary Material 3). The age of the participants ranged from 12 to 89 years and most of the responders allow them to check whether a symptom has previously been reported as an ADR (72%) (Table 3). were women (65%). Nineteen percent of the 656 patients were not prescribed any medicines. Half of the patients 3.3.2 Type of News of Interest indicated they had never used a health app even though they had to be familiar with apps in general to complete the HCPs liked an option to receive news about newly iden- survey, and most were not aware of the Web-RADR app tified drug–drug interactions most (82%), followed by (77%). information about new indications of a drug (75%) (Table 2). They also liked the option to receive news for all 3.2 Outcome Measure: Interest in the App approved marketed drugs (37%). However, the ‘work-/ preference-specific’ answer options (i.e. drugs that they Responders were generally interested in the app for two- way risk communication (Fig. 1). In total, 61% of the prescribe, drugs related to their work and all drugs they are interested in) were together selected by about 60% of the HCPs were very interested in such an app, which ranged from 42% in The Netherlands to 66% in Croatia (Fig. 1a). HCPs (Table 2). HCPs were somewhat more interested in the app than 702 S. T. Vries et al. Fig. 1 a Healthcare 100% Very interested professionals’ interest in an app to report adverse drug reactions 80% (8 responders were excluded; 4 Interested did not complete this question 60% and 4 answered ‘I don’t know’), to receive safety information (5 40% responders were excluded; 4 did Somewhat interested not complete this questions and 20% 1 answered ‘I don’t know’), and for both (i.e. two-way risk communication) (1 responder Not at all interested 0% did not complete this question and was excluded). b Patients’ interest in an app to report adverse drug reactions (15 responders were excluded; 2 did not complete this question and 13 answered ‘I don’t know’), to Total a Netherlands UK receive safety information (14 responders were excluded; 1 did Not interested at all Somewhat interested Interested not complete this question and (on a scale from not interested at all to very interested) Very interested Mean 13 answered ‘I don’t know’), and for both (15 responders a Healthcare professionals were excluded; 3 did not 100% Very interested complete this question and 12 answered ‘I don’t know’). *All European responders. ADRs 80% adverse drug reactions Interested 60% 40% Somewhat interested 20% 0% Not at all interested Total a Netherlands UK Not interested at all Somewhat interested Interested Very interested Mean (on a scale from not interested at all to very interested) Patients Patients liked an option to receive drug safety updates than additional reporting functions (Tables 2, 3). For HCPs (i.e. each newly identified severe ADR of a drug) most this included information about known interactions (84%), followed by newly identified interactions between between drugs (76%), followed by information about how drugs (71%) (Table 3). Only 6% of the patients liked an to resolve an ADR (75%) and an overview of alternative option to receive news about all marketed drugs. drugs to the one for which an ADR is experienced (71%). Most patients liked an overview of ADRs previously 3.3.3 Interest in Other Functions in the App reported by others (73%) and patient information leaflets (72%). Many HCPs and patients selected additional functions that they would like in an app for two-way risk communication. Additional information functions were more often selected Interest in an App for Two-Way Risk Communication 703 Table 2 Healthcare professionals’ preferences and perceptions towards various characteristics of the app Total Croatia The Netherlands UK P-value Perceived benefits in using the app b,c What benefits are there for you in using this app? Faster way to report 317 (83) 152 (80) 51 (86) 62 (84) 0.532 Easier access to ADR report form 276 (73) 131 (69) 39 (66) 61 (82) 0.059 Continue report at a later stage 232 (61) 124 (66) 29 (49) 49 (66) 0.058 Upload a photo 203 (53) 91 (48) 34 (58) 40 (54) 0.383 Store previously reported ADRs 200 (53) 115 (61) 15 (25) 37 (50) \0.001 HR–NL:\ 0.001 NL–UK: 0.004 HR–UK: 0.109 Complete report offline and send it later 179 (47) 90 (48) 19 (32) 39 (53) 0.048 HR–NL: 0.037 NL–UK: 0.018 HR–UK: 0.458 c,d What are the likely benefits in using an app of the \national medicines agency[ to receive safety information? It will keep me up-to-date 318 (84) 159 (83) 45 (80) 68 (88) 0.429 Increased knowledge 290 (76) 155 (81) 35 (63) 58 (75) 0.014 HR–NL: 0.004 NL–UK: 0.111 HR–UK: 0.285 Check whether symptom has been reported as ADR 248 (65) 125 (65) 33 (59) 55 (71) 0.322 Possibility to receive notifications 242 (64) 112 (59) 32 (57) 58 (75) 0.026 HR–NL: 0.842 NL–UK: 0.027 HR–UK: 0.010 Select medicine of interest 180 (47) 84 (44) 19 (34) 42 (55) 0.058 Type of news of interest e,f What type of news about medicines would be useful to you in an app? Newly identified drug–drug interactions 314 (82) 166 (87) 40 (68) 62 (79) 0.003 HR–NL: 0.001 NL–UK: 0.120 HR–UK: 0.124 New indications of a drug 286 (75) 165 (86) 28 (47) 51 (65) \ 0.001 HR–NL:\ 0.001 NL–UK: 0.035 HR–UK:\ 0.001 NCA communications 273 (71) 139 (73) 32 (54) 60 (77) 0.009 HR–NL: 0.007 NL–UK: 0.005 HR–UK: 0.482 Drugs that are taken off the market 262 (68) 126 (66) 37 (63) 58 (74) 0.288 Changes in the PIL 247 (64) 130 (68) 27 (46) 54 (69) 0.004 HR–NL: 0.002 NL–UK: 0.006 HR–UK: 0.852 704 S. T. Vries et al. Table 2 continued Total Croatia The Netherlands UK P-value DHPCs 245 (64) 135 (71) 34 (58) 42 (54) 0.016 HR–NL: 0.061 NL–UK: 0.659 HR–UK: 0.008 Educational materials 186 (49) 106 (56) 17 (29) 32 (41) 0.001 HR–NL:\ 0.001 NL–UK: 0.140 HR–UK: 0.031 Whether re-assessment is ongoing 162 (42) 74 (39) 16 (27) 41 (53) 0.009 HR–NL: 0.104 NL–UK: 0.003 HR–UK: 0.038 Discontinuation of black triangle 149 (39) 77 (40) 9 (15) 46 (59) \ 0.001 HR–NL:\ 0.001 NL–UK:\ 0.001 HR–UK: 0.005 g,h For which medicines would you like to receive news? All approved marketed drugs 137 (37) 61 (32) 20 (35) 33 (45) 0.001 All drugs I am interested in 72 (19) 44 (23) 8 (14) 9 (12) HR–NL: 0.001 Drugs related to my work 98 (26) 60 (32) 10 (18) 15 (20) NL–UK: 0.534 Drugs that I prescribe 66 (18) 23 (12) 19 (33) 17 (23) HR–UK: 0.007 Interest in other functions in the app Please think about an app that you can use for both reporting ADRs and receiving safety information. Which other information functions would f,i you like in such an app? Interactions between drugs 303 (76) 176 (92) 34 (56) 50 (61) \ 0.001 HR–NL:\ 0.001 NL–UK: 0.529 HR–UK:\ 0.001 How to resolve an ADR 296 (75) 148 (77) 42 (69) 57 (70) 0.270 Alternative drugs to the one causing the ADR 282 (71) 139 (72) 43 (70) 54 (66) 0.554 Drug product information 269 (68) 144 (75) 30 (49) 53 (65) 0.001 HR–NL:\ 0.001 NL–UK: 0.064 HR–UK: 0.080 Overview of previously reported ADRs 251 (63) 110 (57) 43 (70) 52 (63) 0.163 Prediction model 218 (55) 99 (52) 30 (49) 49 (60) 0.365 Quality alerts 154 (39) 72 (38) 15 (25) 41 (50) 0.008 HR–NL: 0.064 NL–UK: 0.002 HR–UK: 0.054 f,j Which other reporting functions would you like in such an app? When the medicine cannot be dispensed 231 (68) 124 (72) 29 (64) 46 (66) 0.460 Medicine defects 216 (64) 105 (61) 23 (51) 50 (71) 0.083 Medication errors 205 (60) 106 (62) 18 (40) 44 (63) 0.023 HR–NL: 0.009 NL–UK: 0.016 HR–UK: 0.858 Interest in an App for Two-Way Risk Communication 705 Table 2 continued Total Croatia The Netherlands UK P-value Protection of the app k,l How should an app for you to report ADRs and receive safety information of medicines be protected? Entering an email address and password 109 (30) 63 (34) 11 (20) 21 (29) 0.152 Automatic login 257 (70) 122 (66) 43 (80) 51 (71) Intention to download the app How likely are you to download a free, limited space-taking app to report ADRs and receive safety information of medicines on your device? Not at all likely 9 (2) 1 (1) 1 (2) 6 (7) \ 0.001 Slightly likely 32 (8) 6 (3) 13 (21) 8 (10) HR–NL:\ 0.001 Moderately likely 82 (21) 35 (18) 15 (24) 19 (23) NL–UK: 0.122 Very likely 276 (69) 150 (78) 33 (53) 50 (60) HR–UK:\ 0.001 Data are given as n (%) ADRs adverse drug reactions, DHPCs direct healthcare professional communications, HR Croatia, NCA national competent authority, NL Netherlands, PIL patient information leaflet All European responders The number of responders that do not want to use an app to report ADRs was 12 and the number of responders that selected ‘none’ was 7. 19 responders selected ‘Other’ Percentages are calculated excluding the responders who selected ‘None’ and those who selected ‘I do not want to use an app to report ADRs/ receive safety information’ 6 responders selected ‘None’, 13 responders selected ‘I do not want to use an app to receive safety information’ and 6 selected ‘Other’ 16 responders indicated that they did not want to receive safety information through an app and 10 selected ‘Other’ Percentages are calculated excluding those who selected ‘No other information/reporting functions’ or ‘I do not want to receive safety information through an app’ 15 responders indicated ‘None’ and 11 responders selected ‘Other’ Percentages are calculated excluding those who selected ‘None’ and ‘Other’ 2 responders were not interested in any other type of information function and 37 selected ‘Other’ The number of responders not interested in any other type of reporting function was 59 and 20 selected ‘Other’ One responder did not answer this question and 32 responders selected ‘Other’ Percentages are calculated excluding the responders who selected ‘Other’ 3.3.4 Protection of the App HCPs from The Netherlands and The UK (Table 2). HCPs from The Netherlands generally had a more negative view. Most of the HCPs preferred to use an app for two-way risk HCPs from The UK were more positive to receive news in communication via an automatic login after entering their the app about discontinuation of a black triangle for a drug e-mail address and password once (70%) (Table 2). (59 vs. 40% in Croatia and 15% in The Netherlands; Although most patients also prefer an automatic login, this overall P\0.001) and to receive quality alerts (50 vs. 38% preference was less pronounced (57%) (Table 3). in Croatia and 25% in The Netherlands; overall P = 0.008). 3.3.5 Intention to Download the App Patients from The UK had more positive views on various benefits of using the app than patients from Croatia In total, 69% of the HCPs and 52% of the patients indicated and The Netherlands (Table 3). Patients from The that it is very likely that they will download the app. Only Netherlands were more negative, particularly regarding 2% of the HCPs (Table 2) and 6% of the patients (Table 3) potential benefits of continuing an unfinished report at a indicated that this is not likely at all. later moment (33 vs. 48% in Croatia and 56% in The UK; overall P = 0.001) and of not having to contact a HCP for 3.3.6 Countries in Which the App was Already Available every symptom they experience (24 vs. 50% in Croatia and 53% in The UK; overall P\0.001). In addition, they were HCPs from Croatia appeared to have more positive views less positive about patient information leaflets (61 vs. 76% on potential benefits of the app, the addition of other in both Croatia and The UK; overall P = 0.004), infor- functionalities and the intention to download the app than mation on where to get help (36 vs. 63% in Croatia and 706 S. T. Vries et al. Table 3 Patients’ preferences and perceptions towards various characteristics of the app Total Croatia The Netherlands UK P-value Perceived benefits in using the app b,c What are the likely benefits for you in using this app? Faster way to report 513 (85) 111 (85) 138 (85) 76 (81) 0.592 Easier access to report form 434 (72) 94 (72) 103 (64) 75 (80) 0.020 HR–NL: 0.114 NL–UK: 0.007 HR–UK: 0.199 Store previous reports 290 (48) 63 (48) 70 (43) 46 (49) 0.569 Upload a photo 288 (47) 66 (51) 63 (39) 51 (54) 0.030 HR–NL: 0.042 NL–UK: 0.017 HR–UK: 0.606 Continue a report at a later moment 264 (43) 62 (48) 53 (33) 53 (56) 0.001 HR–NL: 0.009 NL–UK:\ 0.001 HR–UK: 0.199 Complete report offline and send it later 262 (43) 53 (41) 54 (33) 50 (53) 0.008 HR–NL: 0.190 NL–UK: 0.002 HR–UK: 0.066 c,d What are the likely benefits for you in using an app of the\national medicines agency[to receive safety information? Check whether symptom has been reported as ADR 441 (72) 89 (69) 106 (65) 73 (79) 0.048 HR–NL: 0.433 NL–UK: 0.014 HR–UK: 0.086 Increased knowledge 388 (63) 84 (65) 90 (55) 62 (67) 0.079 It will keep me up-to-date 378 (62) 60 (47) 112 (68) 66 (72) \ 0.001 HR–NL:\ 0.001 NL–UK: 0.565 HR–UK:\ 0.001 Possibility to receive notifications 341 (56) 42 (33) 84 (51) 62 (67) \ 0.001 HR–NL: 0.001 NL–UK: 0.012 HR–UK:\ 0.001 Select medicines of interest 311 (51) 51 (40) 70 (43) 52 (57) 0.033 HR–NL: 0.587 NL–UK: 0.033 HR–UK: 0.013 Increased confidence when talking to my HCP 283 (46) 50 (39) 75 (46) 53 (58) 0.021 HR–NL: 0.231 NL–UK: 0.068 HR–UK: 0.006 No need to contact HCP for every symptom 247 (40) 65 (50) 40 (24) 49 (53) \ 0.001 HR–NL:\ 0.001 NL–UK:\ 0.001 HR–UK: 0.674 Type of news of interest e,j What type of news about medicines would be useful to you in an app? Interest in an App for Two-Way Risk Communication 707 Table 3 continued Total Croatia The Netherlands UK P-value Safety updates 520 (84) 104 (78) 139 (84) 77 (85) 0.317 Newly identified drug interactions 439 (71) 85 (64) 105 (64) 79 (87) \0.001 HR–NL: 0.961 NL–UK:\ 0.001 HR–UK: 2009\ 0.001 Changes in the PIL 421 (68) 84 (63) 100 (61) 71 (78) 0.015 HR–NL: 0.652 NL–UK: 0.005 HR–UK: 0.018 New approved used of a drug 324 (53) 69 (52) 69 (42) 43 (47) 0.221 Whether drug review is ongoing 310 (50) 57 (43) 64 (39) 55 (60) 0.003 HR–NL: 0.477 NL–UK: 0.001 HR–UK: 0.010 News on how to take/store the drug 305 (49) 72 (54) 81 (49) 33 (36) 0.029 HR–NL: 0.386 NL–UK: 0.048 HR–UK: 0.008 Experiences of other users of the drug 292 (47) 75 (56) 79 (48) 33 (36) 0.012 HR–NL: 0.144 NL–UK: 0.073 HR–UK: 0.003 Drugs that are temporarily out of stock 243 (39) 45 (34) 65 (39) 28 (31) 0.342 g,h For which medicines would you like to receive news? All approved marketed drugs 36 (6) 10 (8) 8 (5) 7 (8) \0.001 All drugs I am interested in 161 (27) 52 (40) 20 (12) 20 (22) HR–NL:\ 0.001 All drugs to treat my disease 167 (28) 35 (27) 42 (26) 17 (19) NL–UK: 0.112 Drugs prescribed to me 241 (40) 33 (25) 93 (57) 46 (51) HR–UK: 0.001 Interest in other functions in the app Please think about an app that you can use for both reporting side effects and receiving safety information. Which other functions would you i,j like in such an app? Overview of ADRs previously reported 458 (73) 94 (70) 116 (67) 65 (68) 0.846 PIL 450 (72) 102 (76) 105 (61) 72 (76) 0.004 HR–NL: 0.004 NL–UK: 0.013 HR–UK: 0.954 Store list of medicines 405 (64) 71 (53) 102 (59) 71 (75) 0.003 HR–NL: 0.295 NL–UK: 0.010 HR–UK: 0.001 Information on where to get help 368 (59) 84 (63) 63 (36) 56 (59) \0.001 HR–NL:\ 0.001 NL–UK:\ 0.001 HR–UK: 0.567 Reminder to take medicines 321 (51) 79 (59) 54 (31) 54 (57) \0.001 HR–NL:\ 0.001 NL–UK:\ 0.001 HR–UK: 0.750 Option to report medicine defects 321 (51) 69 (51) 83 (48) 53 (56) 0.468 708 S. T. Vries et al. Table 3 continued Total Croatia The Netherlands UK P-value Learn from other patient experiences 299 (48) 77 (57) 73 (42) 50 (53) 0.024 HR–NL: 0.008 NL–UK: 0.101 HR–UK: 0.469 Chat with others about own experiences 140 (22) 41 (31) 24 (14) 26 (27) 0.001 HR–NL:\ 0.001 NL–UK: 0.007 HR–UK: 0.597 Protection of the app f,k How should an app for you to report side effects and receive safety information of medicines be protected? Entering an email address and password 255 (43) 50 (39) 58 (38) 40 (44) 0.606 Automatic login 332 (57) 79 (61) 96 (62) 51 (56) Intention to download the app How likely are you to download a free, limited space taking app to report side-effects and receive safety information of medicines on your device? Not at all likely 37 (6) 4 (3) 19 (10) 7 (7) 0.001 Slightly likely 87 (13) 11 (8) 41 (22) 17 (17) HR–NL:\ 0.001 Moderately likely 189 (29) 54 (40) 44 (24) 27 (27) NL–UK: 0.572 Very likely 341 (52) 67 (49) 83 (44) 48 (48) HR–UK: 0.035 Data are given as n (%) ADRs adverse drug reactions, DHPCs direct healthcare professional communications, HR Croatia, NCA national competent authority, NL Netherlands, PIL patient information leaflet All European responders The number of responders who do not want to use an app to report ADRs was 22, the number of responders that selected ‘none’ was 27 and 36 selected ‘Other’ Percentages are calculated excluding the responders who selected ‘None’ and those who selected ‘I do not want to use an app to report side effects/receive safety information’ 24 responders selected ‘None’, 20 responders selected ‘I do not want to use an app to receive safety information’ and 15 selected ‘Other’ 39 responders indicated that they do not want to receive safety information through an app and 29 selected ‘Other’ Percentages are calculated excluding the responders who selected ‘Other’ 27 responders indicated ‘Not applicable’ and 24 responders selected ‘Other’ Percentages are calculated excluding those who selected ‘Not applicable’ and ‘Other’ 1 responder did not complete this question, 26 were not interested in any other function and 64 selected ‘Other’ Percentages are calculated excluding those who selected ‘None/No other functions/I do not want to receive safety information through an app’ 4 responders did not answer this question and 65 responders selected ‘Other’ 2 responders did not complete this question 59% in The UK; overall P\0.001), a reminder to take characteristics and being interested in the app (Electronic medicines (31 vs. 59% in Croatia and 57% in The UK; Supplementary Material 4). overall P\0.001), and a functionality to chat with others about their own experiences (14 vs. 31% in Croatia and 3.4.1 HCP Characteristics 27% in The UK; overall P = 0.001). Of the four determinants included in the analyses of the 3.4 Determinants: HCP/Patient Characteristics HCPs, only the use of a health app was significantly associated with interest in the app. HCPs who at least Of the 399 HCPs and 656 patients who completed the sometimes use a health app were more often very interested survey, 390 and 636, respectively, were included in the than those who never use such an app (odds ratio [OR] analyses to assess the association between HCP/patient 3.52; 95% confidence interval [CI] 1.96–6.30) (Fig. 2a). The sensitivity analyses per country of interest showed that Interest in an App for Two-Way Risk Communication 709 this HCP characteristic was statistically significant for The patients were less often very interested in the app than UK only (OR 9.50; 95% CI 3.11–29.05) (Electronic Sup- younger patients (OR 0.98; 95% CI 0.97–0.997). Patients plementary Material 5). The generalised ordered logit who at least sometimes use a health app were more often model showed a similar influence of the use of health apps very interested than those who never use a health app (OR on the different levels of the outcome measure (Electronic 1.64; 95% CI 1.19–2.27) (Fig. 2b). The country-specific Supplementary Material 6). analyses showed a statistically significant association of the use of health apps for The Netherlands only (OR 2.20; 95% 3.4.2 Patient Characteristics CI 1.13–4.27) (Electronic Supplementary Material 7). Additional statistically significant associations were shown Age and use of health apps were the patient characteristics for The UK, where patients with a tertiary education level significantly associated with interest in the app. Older were less often very interested than patients with a low or N = 390 Determinants Odd raos (95% CIs) Age 1.00 (0.98-1.02) Female vs. Male 1.20 (0.76-1.89) At least sometimes use At least sometimes use of health app vs. Never 3.52 (1.96-6.30) of health apps vs. Never ADR ADR reported to agency reported to agency vs. Not 1.52 (0.94-2.47) vs. Not 0,01 0,1 1 10 100 Odds raos with 95% confidence intervals (log scale) a Healthcare professionals N = 636 Determinants Odd raos (95% CIs) Age 0.98 (0.97-0.997) Female vs. Male 1.14 (0.81-1.62) Tertiary education vs. Tertiary education vs. Low/secondary 0.75 (0.53-1.05) Low/secondary 0 medicines vs. 1-4 0.69 (0.43-1.09) ≥5 medicines vs. 1-4 1.02 (0.70-1.48) At least sometimes use 1.64 (1.19-2.27) At least sometimes use of health apps vs. Never of health apps vs. Never ADR experienced vs. Not 1.47 (0.97-2.21) Aware of reporting to Aware of reporting to agency vs. Not 1.17 (0.84-1.62) agency vs. Not 0,01 0,1 1 10 100 Odds raos with 95% confidence intervals (log scale) b Paents Fig. 2 Odds ratios with 95% confidence intervals of associations characteristics and being very interested in this app. ADR adverse between a healthcare professional characteristics and being very drug reaction, CI confidence interval interested in an app for two-way risk communication and b patient 710 S. T. Vries et al. secondary education level (OR 0.20; 95% CI 0.05–0.81), study showed that one-third of patients did not discuss their and patients who take no medicines were less often very medication symptoms with an HCP [23]. Almost half of the interested than patients who take one to four medicines patients in our study indicated that the app could increase (OR 0.17; 95% CI 0.04–0.75). The generalised ordered their confidence when talking to their HCP. This increased logit model showed a similar effect of use of health apps on certainty about an ADR may improve the patient–HCP the different levels of the outcome measure but showed conversation about ADRs. In addition, it may increase different patterns for other characteristics (Electronic patient reporting of ADRs to the national pharmacovigi- Supplementary Material 6). lance centres, but future studies will be needed to investi- gate such effects. HCPs’ and patients’ preferences and perceptions 4 Discussion towards the characteristics of the app in general were rel- atively similar. This suggests that the functionality of the This study showed that HCPs and patients were generally app can be similar for HCPs and patients. However, the interested in an app for two-way risk communication and type of drugs for which responders would like to receive that, in particular, HCPs and patients who already use a safety information differed between HCPs and patients, as health app are more interested in such an app. A main HCPs liked the option to receive information for all drugs benefit for HCPs and patients of the reporting functionality more. It should be possible to incorporate such user- in the app was that it can make the ADR reporting process specific preferences into the app. In addition, differences in faster and easier. Previous studies have shown that lack of other aspects of the app, such as appropriate terminology time and difficulty in accessing reporting forms are the for these target groups, need to be considered [2]. main barriers for HCPs to spontaneously report ADRs to Our finding that HCPs and patients who already use a national pharmacovigilance centres [12–14]. In contrast, health app are particularly interested in an app for two-way literature suggests that patients may be willing to spend risk communication suggests that, in particular, those HCPs more time on reporting an ADR than HCPs [2] and one of and patients may well be the most receptive group that the reasons for patients to spontaneously report an ADR is should be informed about the existence of such an app. when they have the impression that HCPs have too limited This could, for instance, be done via advertisements or a time to accurately report ADRs [15]. However, in this link to the app in other health apps. The high number of study patients also preferred a reporting tool that is easy responders that liked other functionalities in an app for and fast to complete. two-way risk communication also suggests that links to With respect to the information functionality of the app, various other health apps may increase its usefulness. Ways to stimulate the interest of non-app users and encouraging HCPs would like the app to keep them up-to-date with the latest news and that it will increase their knowledge about their participation could benefit from further investigation. drug safety. In addition, most HCPs liked to receive news about (newly identified) drug–drug interactions. Previ- 4.1 Strengths and Limitations ously, it has been shown that HCPs’ awareness of drug safety issues, for instance those communicated through A strength of this study is the assessment of interest in the direct healthcare professional communications (DHPCs), app among both HCPs and patients. In addition, this is a are suboptimal [16] as is their knowledge about drug–drug first study assessing the role of user characteristics on the interactions [17]. Our study suggests that HCPs are aware use of an app for two-way risk communication. Although of their lack of knowledge and that an app could be a tool we collected data from a large sample of HCPs and to improve this. patients, a limitation is that the number of responders per Patients particularly liked an overview of ADRs previ- country was still relatively low. Another limitation of this ously reported by others, and patient information leaflets as study was its methodology of a cross-sectional survey. We information in an app. They indicated that a main benefit of cannot be sure how representative it was of the studied the app would be to allow them to check whether a countries. We present data for a subgroup of three countries symptom was previously reported as an ADR. These in which the app was already available, but differences findings are in line with previous studies showing that between countries should be interpreted cautiously since patients are sometimes uncertain about an association the characteristics of the responders differ across these between a symptom and a drug [18–21], and may be countries. Moreover, the number of responders from uncertain about the exact drug causing the symptoms [22]. countries other than the three countries in which the app Providing such information in an app could reduce was available was low. Therefore, it cannot be assumed patients’ uncertainty in confirming that their symptoms are that the included population is a representative sample of caused by the drug(s) they are taking. Also, a previous the European populations. Also, we do not have any Interest in an App for Two-Way Risk Communication 711 Seventh Framework Programme FP7/2007-2013 and EFPIA compa- numbers relating to response rates since HCPs and patients nies’ in kind contribution, http://www.imi.europa.eu, UK. were reached via various channels including advertise- ments and announcements. Furthermore, survey-answering Conflict of interest Sieta T. de Vries, Petra Denig, Carmen Lasheras tendencies may differ across countries, as has been shown Ruiz, Franc ¸ois Houy ¨ ez, Lisa Wong, and Alastair Sutcliffe have no conflicts of interest that are directly relevant to the content of this previously [24]. Another limitation relates to the assess- study. Peter G. M. Mol is an employee of the Dutch Medicines ment of interest in the app. First, we could not use a val- Evaluation Board. idated measure to assess interest in the app since, to our knowledge, such a measure is not available. Second, Open Access This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International interest is a first step for actually downloading and using License (http://creativecommons.org/licenses/by-nc/4.0/), which per- the app but its actual use may be influenced by other factors mits any noncommercial use, distribution, and reproduction in any [3]. Therefore, future studies are needed to evaluate the medium, provided you give appropriate credit to the original actual use of the app in different countries. author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. 5 Conclusions References HCPs and patients in Europe are generally interested in an app for two-way risk communication, which supports its 1. Ghosh R, Lewis D. Aims and approaches of Web-RADR: a further development. Such an app should support easy and consortium ensuring reliable ADR reporting via mobile devices fast reporting of ADRs and provide information about and new insights from social media. Expert Opin Drug Saf. 2015;14(12):1845–53. drug–drug interactions and previously reported ADRs to its 2. de Vries ST, Wong L, Sutcliffe A, Houyez F, Ruiz CL, Mol PG, users. HCPs and patients who already use other health apps et al. Factors influencing the use of a mobile app for reporting are particularly interested in the app. Therefore, dissemi- adverse drug reactions and receiving safety information: a qual- nation strategies could focus on reaching these HCPs and itative study. Drug Saf. 2017;40(5):443–55. 3. Venkatesh V, Morris MG, Gordon B, Davis FD. User acceptance patients. of information technology: toward a unified view. MIS Q. Acknowledgements The healthcare professionals and patients who 2003;27(3):425–78. responded to the survey are greatly appreciated. We acknowledge 4. Ali EE, Leow JL, Chew L, Yap KY. Patients’ perception of app- other Web-RADR Work Package 3b members: Raphael van Eemeren, based educational and behavioural interventions for enhancing Karin Hace, Sandra Fernandes, Faiza Afzal and Denis Costello. oral anticancer medication adherence. J Cancer Educ. https://doi. org/10.1007/s13187-017-1248-x (Epub 2017 Jul 14). Author Contributions All authors contributed to the development 5. Lucero RJ, Frimpong JA, Fehlberg EA, Bjarnadottir RI, Weaver and formulation of the research question. All authors were involved in MT, Cook C, et al. The relationship between individual charac- distribution of the survey. STdV, PD, and PGMM conducted the teristics and interest in using a mobile phone app for HIV self- analyses. All authors contributed to the interpretation of the data. management: observational cohort study of people living with STdV wrote the manuscript. PD, CLR, FH, LW, AS, and PGMM HIV. JMIR Mhealth Uhealth. 2017;5(7):e100. reviewed and edited the manuscript. All authors have read and 6. Bender MS, Choi J, Arai S, Paul SM, Gonzalez P, Fukuoka Y. Digital technology ownership, usage, and factors predicting approved the final manuscript. downloading health apps among Caucasian, Filipino, Korean, and Latino Americans: the digital link to health survey. JMIR Compliance with Ethical Standards Mhealth Uhealth. 2014;2(4):e43. 7. Carroll JK, Moorhead A, Bond R, LeBlanc WG, Petrella RJ, Ethical approval The study protocol was submitted to (a member of) Fiscella K. Who uses mobile phone health apps and does use an ethics committee in The UK, The Netherlands and Germany. In matter? A secondary data analytics approach. J Med Internet Res. The UK, approval was obtained from the UCL Research Ethics 2017;19(4):e125. Committee (Project ID number 6855/001). In The Netherlands, the 8. Krebs P, Duncan DT. Health app use among US mobile phone Medical Ethics Committee of the University Medical Center owners: a national survey. JMIR Mhealth Uhealth. Groningen (METc UMCG) determined that ethical approval was not 2015;3(4):e101. needed for this study (reference number M16.191043). In Germany, a 9. Montastruc F, Bagheri H, Lacroix I, Damase-Michel C, Chebane review by an ethics committee was not necessary. Therefore, no L, Rousseau V, et al. Adverse drug reaction reports received further approval was deemed necessary for the other countries. All through the mobile app, VigiBIP: a comparison with classical procedures performed in studies involving human participants were in methods of reporting. Drug Saf. https://doi.org/10.1007/s40264- accordance with the ethical standards of the institutional and/or 017-0630-2 (Epub 2017 Dec 21). national research committee and with the 1964 Helsinki Declaration 10. Williams R. Generalized ordered logit/partial proportional odds and its later amendments or comparable ethical standards. models for ordinal dependent variables. Stata J. 2006;6(1):58–82. 11. Williams R. Understanding and interpreting generalized ordered Funding The Web-RADR project has received support from the logit models. J Math Sociol. 2016;40(1):7–20. Innovative Medicine Initiative Joint Undertaking (http://www.imi. 12. Vallano A, Cereza G, Pedros C, Agusti A, Danes I, Aguilera C, europa.eu) under Grant Agreement n Æ 115632, resources of which are et al. Obstacles and solutions for spontaneous reporting of composed of financial contribution from the European Union’s 712 S. T. Vries et al. adverse drug reactions in the hospital. Br J Clin Pharmacol. and assess symptoms as adverse drug reactions. Eur J Clin 2005;60(6):653–8. Pharmacol. 2014;70(5):607–15. 13. Lopez-Gonzalez E, Herdeiro MT, Figueiras A. Determinants of 19. de Vries ST, Mol PG, de Zeeuw D, Haaijer-Ruskamp FM, Denig under-reporting of adverse drug reactions: a systematic review. P. Development and initial validation of a patient-reported Drug Saf. 2009;32(1):19–31. adverse drug event questionnaire. Drug Saf. 2013;36(9):765–77. 14. De Angelis A, Colaceci S, Giusti A, Vellone E, Alvaro R. Factors 20. de Vries ST, Haaijer-Ruskamp FM, de Zeeuw D, Denig P. The that condition the spontaneous reporting of adverse drug reactions validity of a patient-reported adverse drug event questionnaire among nurses: an integrative review. J Nurs Manag. using different recall periods. Qual Life Res. 2016;24(2):151–63. 2014;23(9):2439–45. 15. Al Dweik R, Stacey D, Kohen D, Yaya S. Factors affecting 21. Britten N. Medication errors: the role of the patient. Br J Clin patient reporting of adverse drug reactions: a systematic review. Pharmacol. 2009;67(6):646–50. Br J Clin Pharmacol. 2017;83(4):875–83. 22. de Vries ST, Haaijer-Ruskamp FM, de Zeeuw D, Denig P. 16. Piening S, Haaijer-Ruskamp FM, de Graeff PA, Straus SM, Mol Construct and concurrent validity of a patient-reported adverse PG. Healthcare professionals’ self-reported experiences and drug event questionnaire: a cross-sectional study. Health Qual preferences related to direct healthcare professional communi- Life Outcomes. 2014;12:103. cations: a survey conducted in the Netherlands. Drug Saf. 23. Weingart SN, Gandhi TK, Seger AC, Seger DL, Borus J, Burdick 2012;35(11):1061–72. E, et al. Patient-reported medication symptoms in primary care. 17. Glassman PA, Simon B, Belperio P, Lanto A. Improving recog- Arch Intern Med. 2005;165(2):234–40. nition of drug interactions: benefits and barriers to using auto- 24. Tellis GJ, Chandrasekaran D. Does culture matter? Assessing mated drug alerts. Med Care. 2002;40(12):1161–71. response biases in cross-national survey research. Int J Res Mark 18. Chaipichit N, Krska J, Pratipanawatr T, Uchaipichat V, Jarern- 2010. https://ssrn.com/abstract=1659911. Accessed 30 Jan 2018. siripornkul N. A qualitative study to explore how patients identify http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Drug Safety Springer Journals

Interest in a Mobile App for Two-Way Risk Communication: A Survey Study Among European Healthcare Professionals and Patients

Download PDF
16 pages

Loading next page...
 
/lp/springer_journal/interest-in-a-mobile-app-for-two-way-risk-communication-a-survey-study-bNfJhj0c0b

References (30)

Publisher
Springer Journals
Copyright
Copyright © 2018 by The Author(s)
Subject
Medicine & Public Health; Drug Safety and Pharmacovigilance; Pharmacology/Toxicology
ISSN
0114-5916
eISSN
1179-1942
DOI
10.1007/s40264-018-0648-0
pmid
29500800
Publisher site
See Article on Publisher Site

Abstract

Drug Saf (2018) 41:697–712 https://doi.org/10.1007/s40264-018-0648-0 OR IGINAL RESEARCH ARTIC L E Interest in a Mobile App for Two-Way Risk Communication: A Survey Study Among European Healthcare Professionals and Patients 1 1 2 2 • • • • Sieta T. de Vries Petra Denig Carmen Lasheras Ruiz Franc ¸ois Houy ¨ ez 3 3 1 • • • Lisa Wong Alastair Sutcliffe Peter G. M. Mol on behalf of IMI Web-RADR Work Package 3b Consortium Published online: 2 March 2018 The Author(s) 2018. This article is an open access publication Abstract characteristics on the level of interest in the app (i.e. very Introduction Previously, an app has been developed for interested vs. not/somewhat interested). healthcare professionals (HCPs) and patients to report Results In total, 399 HCPs and 656 patients completed the adverse drug reactions (ADRs) to national medicines survey. About half of the patients (48%; ranging from 38% agencies and to receive drug safety information. from The Netherlands to 54% from The UK), and 61% of the Objective This study aimed to assess (1) European HCPs’ and HCPs (ranging from 42% from The Netherlands to 54% from patients’ interest in an app for this two-way risk communica- The UK) were very interested in the app. A faster means of tion; (2) their preferences and perceptions towards specific app reporting ADRs and easier access to the reporting form were characteristics; and (3) which HCPs and patients are particu- the main perceived benefits. HCPs and patients who already larly interested in the app. In addition, these aspects were use a health app were particularly interested in the app studied specifically for the countries where such an app was (HCPs: odds ratio [OR] 3.52; 95% confidence interval [CI] already available, i.e. Croatia, The Netherlands, and The UK. 1.96–6.30, patients: OR 1.64; 95% CI 1.19–2.27). Methods European HCPs and patients were asked to Conclusions An app is positively perceived by HCPs and complete a web-based survey developed in the context of patients for reporting ADRs quickly and for receiving drug the Web-Recognizing Adverse Drug Reactions (Web- safety information from national medicines agencies. In RADR) project. Data on app interest and preferences and particular, HCPs and patients who already use other health perceptions towards app characteristics were analysed apps were interested in the app. descriptively. Logistic regression analyses were conducted to assess the association of HCP characteristics and patient Key Points The members of IMI Web-RADR Work Package 3b Consortium are Interest in an app for two-way risk communication listed in acknowledgements. (i.e. to report adverse drug reactions [ADRs] to national medicines agencies and to receive drug Electronic supplementary material The online version of this article (https://doi.org/10.1007/s40264-018-0648-0) contains supple- safety information) is high among healthcare mentary material, which is available to authorized users. professionals (HCPs) and patients. & Peter G. M. Mol The app should be a faster way to report ADRs than [email protected] conventional reporting options and should preferably offer additional information about drug–drug Department of Clinical Pharmacy and Pharmacology, interactions and previously reported ADRs. University of Groningen, University Medical Center Groningen, Groningen, The Netherlands Strategies to disseminate an app on two-way risk European Organisation for Rare Diseases (Eurordis), Paris, communication could focus on targeting HCPs and France patients who already use a health app since these Population, Policy and Practice Programme, UCL Institute of persons were particularly interested in the app. Child Health, 30 Guilford Street, London WC1N 1EH, UK 698 S. T. Vries et al. The aim of the current study was to assess (1) European 1 Introduction HCPs’ and patients’ interest in an app for two-way risk communication; (2) their preferences and perceptions Healthcare professionals (HCPs) and patients have access towards specific characteristics of the app; and (3) which to a plethora of health-related mobile apps but not every HCPs and patients are particularly interested in such an person is equally interested in such apps. Recently, a health app. In addition, these aspects were specifically studied for app developed in the context of the Web-Recognizing the countries where such an app was already available, i.e. Adverse Drug Reactions (Web-RADR) project (https:// Croatia, The Netherlands and The UK. This knowledge can web-radr.eu/) was added to the available health-related be used by national medicines agencies in the development apps. The goal of this app is to provide two-way risk or improvement of an app for two-way risk communication communication, defined as the possibility to report adverse and in the development of strategies to inform potential drug reactions (ADRs) to national medicines agencies/ users about the existence of the app. pharmacovigilance centres, and to receive drug safety information from these agencies [1]. Previously, in a qualitative study, we identified various factors that may 2 Methods influence the use of this app and showed that HCPs and patients were generally positive about its development [2]. 2.1 Study Design and Survey Development However, these aspects should be validated in a larger population. In this cross-sectional study, data were collected between Considering the plethora of new technologies, including July and October 2016 using web-based surveys. Two apps, theoretical models have been developed attempting surveys (i.e. one for HCPs and one for patients) were to identify factors that influence the uptake of the new developed in English by members of the Web-RADR technology. An example is the Unified Theory of Accep- project (see Electronic Supplementary Material 1 and 2 for tance and Use of Technology, which states that user the HCP and patient survey, respectively). The English- characteristics play a moderating role in the acceptance of language surveys were translated by an official translation technology [3]. This indicates that not every HCP or agency into Croatian, Dutch, French, German, Portuguese patient will be interested in the app. Several studies have and Spanish. Web-RADR members checked whether the investigated whether interest in health apps is influenced by translations had the same meaning as the English version. characteristics such as age, ethnicity/race, gender, current The web-based format of the surveys was created using use of a health app, inability to work, income, educational Unipark software (http://unipark.com/en/). A separate link degree, clinical characteristics or having a family member was available for the HCP and patient survey in each of the with a specific disease [4–8]. Although the studies con- languages. sistently show that older people are generally less inter- The content of the surveys was based on the results of a ested in health apps than younger people [4–8], the qualitative study [2], input from members of other work literature is inconclusive about other user characteristics. packages of the Web-RADR project, and various HCP and For instance, Latinos/Hispanics were less interested in one patient organisations. The patient survey contained ques- study [6] but more interested in another study [8] than tions about ADR reporting in general; their opinion of an Caucasians/white people. Likewise, one study showed that app to report ADRs, an app to receive safety information, males were slightly less interested than females [7], and an app for two-way risk communication; reporting whereas another study showed no gender differences [8]. ADRs through an app of the national medicines agency; The previous studies conducted in the USA [5–8]or safety information and receiving such information through Asia (i.e. Singapore) [4] focused on patients or the general an app; and, finally, some general questions such as age, population, and assessed a person’s interest in a health app gender and the country in which they lived at the time of in general [6–8] or in apps to support adherence or self- survey completion. management [4, 5]. A recent study about the VigiBIP app, In some questions, the name of the national medicines developed by the Toulouse University Pharmacovigilance agency/pharmacovigilance centre was mentioned. The Center for two-way risk communication, suggests that Agenciju za lijekove i medicinske proizvode (HALMED), patients are interested in the app and that different ADRs Nederlands Bijwerkingencentrum (Lareb), Medicines and may by reported via the app compared with conventional Health Regulatory Agency (MHRA), l’Agence nationale methods [9]. However, more studies on characteristics of de se ´curite ´ du me ´dicament et des produits de sante ´ HCPs and patients on their interest in apps for communi- (ANSM), Bundesinstitutes fur Arzneimittel und Medizin- cating health-related issues with national medicines agen- produkte, Autoridade Nacional do Medicamento e Produ- cies are needed. tos de Sau ´ de (INFARMED) and la Agencia Espan ˜ ola del Interest in an App for Two-Way Risk Communication 699 Medicamento y Productos Sanitarios (AEMPS) were 2.5 Determinants: Healthcare Professional (HCP)/ mentioned in the Croatian, Dutch, English, French, Ger- Patient Characteristics man, Portuguese and Spanish versions of the survey, respectively. The following HCP characteristics were assessed as determinants for HCPs’ interest in an app for two-way risk Participants gave their implied consent to participate in the study by voluntarily completing the survey. communication: age, gender, how often they already used health apps and whether they had ever reported an ADR to 2.2 Participants and Data Collection the national medicines agency. For patients, the following characteristics were assessed: age, gender, educational Any HCP or patient in Europe familiar with mobile apps level, number of medicines, how often they already used was eligible to participate in this study. All HCPs were health apps, whether they had ever experienced an ADR considered to be familiar with apps. Patients were informed and whether they were aware they could report ADRs to that they should only complete the survey if they were the national medicines agency (Table 1). familiar with apps. Various channels were used to reach HCPs and patients. For instance, European and national 2.6 Analyses HCP and patient organisations distributed the survey among members via direct e-mail or advertisements on Descriptive statistics are presented for HCPs and patients separately. In addition, this is presented for countries in their websites and/or in their newsletters. The survey was also announced on Facebook and Twitter accounts, for which the app was already available. Completers of the instance on the account of the Web-RADR project. Croatian, Dutch or English version of the survey who Recruitment strategies focused particularly on reaching indicated they were living in these countries at the time of HCPs and patients in Croatia, The Netherlands and The UK the survey were included in these country-specific analy- since the Web-RADR app on two-way risk communication ses. Differences across these countries were tested using 2 2 was available in these countries at the time of this study. Chi-squared (v ) tests. Three post hoc v -tests were con- The pharmacovigilance centres in Croatia, The Netherlands ducted in the case of P\0.05 to test which countries dif- and The UK also distributed the survey, for instance by fered from each other. The Bonferroni correction was applied for these post hoc analyses to correct for multiple posting a message on their respective websites. To encourage response rates, survey completers had the option testing. This implies that P values\0.016 were considered statistically significant. to participate in a prize draw to win a €50 coupon. Logistic regression analyses were conducted to assess associations between responder characteristics and the 2.3 Outcome Measure: Interest in the App dichotomised outcome measure, expressing a high interest The outcome measure of this study was responders’ interest in the app. For this, being very interested was contrasted in an app for two-way risk communication. Responders with being somewhat or not interested (Table 1). In sen- were asked to rate on a 4-point Likert scale to what extent sitivity analyses using generalised ordered logit models they were interested in such an app (Table 1). Responders [10, 11], we assessed whether this dichotomisation resulted could also indicate that they did not know whether they had in a loss of information. Responders were excluded from the logistic regression analyses and generalised ordered interest in the app. logit models when they (1) selected another answer option than male/female on the question about their gender; (2) 2.4 Determinants: App Characteristics did not answer or answered ‘don’t know’ on the question Expectations and actual characteristics of an app may about the app interest (outcome variable); or (3) did not influence someone’s intention to download and use an app answer a question that was used as a determinant in these [3]. Therefore, responders were asked about their prefer- analyses. ences and perceptions regarding an app for two-way risk All analyses were conducted using Stata version 13 communication. For this, questions were asked about per- (Stata Corp., College Station, TX, USA). Microsoft Excel ceived benefits in using the app, the type of news of 2010 (Microsoft Corp., Redmond, WA, USA) was used for interest, interest in other functions in the app and the the graphical presentation of the results. protection of the app. In addition, responders were asked about their intention to download an app for two-way risk communication. 700 S. T. Vries et al. Table 1 Questions and answer options used as outcome variable and determinants Variable Question Answer options Type of variable in analyses Analyses of HCPs/patients Outcome variable App interest In general, how interested would you be in an app of Not interested at all Dichotomous: not/(somewhat) HCPs and the\national medicines agency[ that you can use interested vs. very interested patients Somewhat for both, reporting side effects/adverse drug interested Don’t know ? excluded reactions and receiving safety information? Interested Very interested Don’t know Determinants Age What is your age? Continuous Continuous HCPs and patients Gender What is your gender? Male Dichotomous: male vs. female HCPs and patients Female Other/prefer not to say ? excluded Other/prefer not to say Educational What is your highest level of education completed? No formal education Dichotomous: low/secondary Patients level or below education vs. tertiary education (first and second Primary education stage) Lower secondary education Upper secondary education Post-secondary but non-tertiary education First stage of tertiary education Second stage of tertiary education Number of How many different medicines are prescribed to you 0 Categorical: 0 medicines; 1–4 Patients medicines at the moment? medicines (reference category); C 5 medicines 5 or more Use health How often do you use a health app? Daily Dichotomous: never vs. other HCPs and apps answer options patients Weekly Monthly or less often Never Experience of Have you ever experienced a side effect of a Yes Dichotomous: no/don’t know Patients ADRs medicine that you take or have taken in the past? vs. yes No Don’t know/don’t remember Awareness of Are you aware that you can report experienced side Yes Dichotomous: no/I have never Patients reporting effects to the \national medicines agency[? heard of the \national No ADRs medicines agency[ vs. yes I have never heard of the \national medicines agency[ Interest in an App for Two-Way Risk Communication 701 Table 1 continued Variable Question Answer options Type of variable in analyses Analyses of HCPs/patients Report ADR Have you ever reported an adverse drug reaction Yes Dichotomous: no/don’t know HCPs to national experienced by your patients to the \national vs. yes No medicines medicines agency[? Don’t know/don’t agency remember ADR adverse drug reaction, HCPs healthcare professionals patients. About half of the patients (48%) were very 3 Results interested in the app, which ranged from 38% in The 3.1 Characteristics of the Responders Netherlands to 54% in The UK (Fig. 1b). Interest in an app for two-way risk communication was somewhat higher 3.1.1 HCPs than interest in an app with single functionality (i.e. reporting of ADRs or receiving safety information). In total, 399 HCPs completed the survey: 192 were from 3.3 Determinants: App Characteristics Croatia, 62 were from The Netherlands, 83 were from The UK and 62 (16%) were from other European countries (i.e. 3.3.1 Perceived Benefits in Using the App countries where the app was not rolled out) (see Electronic Supplementary Material 3). The age of the responders ranged from 20 to 71 years and most of the responders With respect to the reporting functionality of the app, most of the HCPs and patients indicated that a faster way to were women (68%). Sixteen percent of the 399 HCPs indicated they had never used a health app. More than half report ADRs and easier access to the ADR reporting form were potential benefits of using the app. These answer of the HCPs had at least heard about the Web-RADR app. options were selected by 83 and 73% of the HCPs, respectively (Table 2) and by 85 and 72% of patients, 3.1.2 Patients respectively (Table 3). Keeping up-to-date with the latest drug safety news There were 656 patients who completed the survey, of (84%) and increasing their drug safety knowledge (76%) whom 136 were from Croatia, 187 were from The were important benefits perceived by HCPs on using an Netherlands, and 100 were from The UK. The remaining app. The possibility to select medicines of interest was seen 233 (36%) patients were from other European countries as the least beneficial option for HCPs (47%) (Table 2). (i.e. countries where the app was not rolled out) (Electronic Most of the patients saw it as a benefit that the app would Supplementary Material 3). The age of the participants ranged from 12 to 89 years and most of the responders allow them to check whether a symptom has previously been reported as an ADR (72%) (Table 3). were women (65%). Nineteen percent of the 656 patients were not prescribed any medicines. Half of the patients 3.3.2 Type of News of Interest indicated they had never used a health app even though they had to be familiar with apps in general to complete the HCPs liked an option to receive news about newly iden- survey, and most were not aware of the Web-RADR app tified drug–drug interactions most (82%), followed by (77%). information about new indications of a drug (75%) (Table 2). They also liked the option to receive news for all 3.2 Outcome Measure: Interest in the App approved marketed drugs (37%). However, the ‘work-/ preference-specific’ answer options (i.e. drugs that they Responders were generally interested in the app for two- way risk communication (Fig. 1). In total, 61% of the prescribe, drugs related to their work and all drugs they are interested in) were together selected by about 60% of the HCPs were very interested in such an app, which ranged from 42% in The Netherlands to 66% in Croatia (Fig. 1a). HCPs (Table 2). HCPs were somewhat more interested in the app than 702 S. T. Vries et al. Fig. 1 a Healthcare 100% Very interested professionals’ interest in an app to report adverse drug reactions 80% (8 responders were excluded; 4 Interested did not complete this question 60% and 4 answered ‘I don’t know’), to receive safety information (5 40% responders were excluded; 4 did Somewhat interested not complete this questions and 20% 1 answered ‘I don’t know’), and for both (i.e. two-way risk communication) (1 responder Not at all interested 0% did not complete this question and was excluded). b Patients’ interest in an app to report adverse drug reactions (15 responders were excluded; 2 did not complete this question and 13 answered ‘I don’t know’), to Total a Netherlands UK receive safety information (14 responders were excluded; 1 did Not interested at all Somewhat interested Interested not complete this question and (on a scale from not interested at all to very interested) Very interested Mean 13 answered ‘I don’t know’), and for both (15 responders a Healthcare professionals were excluded; 3 did not 100% Very interested complete this question and 12 answered ‘I don’t know’). *All European responders. ADRs 80% adverse drug reactions Interested 60% 40% Somewhat interested 20% 0% Not at all interested Total a Netherlands UK Not interested at all Somewhat interested Interested Very interested Mean (on a scale from not interested at all to very interested) Patients Patients liked an option to receive drug safety updates than additional reporting functions (Tables 2, 3). For HCPs (i.e. each newly identified severe ADR of a drug) most this included information about known interactions (84%), followed by newly identified interactions between between drugs (76%), followed by information about how drugs (71%) (Table 3). Only 6% of the patients liked an to resolve an ADR (75%) and an overview of alternative option to receive news about all marketed drugs. drugs to the one for which an ADR is experienced (71%). Most patients liked an overview of ADRs previously 3.3.3 Interest in Other Functions in the App reported by others (73%) and patient information leaflets (72%). Many HCPs and patients selected additional functions that they would like in an app for two-way risk communication. Additional information functions were more often selected Interest in an App for Two-Way Risk Communication 703 Table 2 Healthcare professionals’ preferences and perceptions towards various characteristics of the app Total Croatia The Netherlands UK P-value Perceived benefits in using the app b,c What benefits are there for you in using this app? Faster way to report 317 (83) 152 (80) 51 (86) 62 (84) 0.532 Easier access to ADR report form 276 (73) 131 (69) 39 (66) 61 (82) 0.059 Continue report at a later stage 232 (61) 124 (66) 29 (49) 49 (66) 0.058 Upload a photo 203 (53) 91 (48) 34 (58) 40 (54) 0.383 Store previously reported ADRs 200 (53) 115 (61) 15 (25) 37 (50) \0.001 HR–NL:\ 0.001 NL–UK: 0.004 HR–UK: 0.109 Complete report offline and send it later 179 (47) 90 (48) 19 (32) 39 (53) 0.048 HR–NL: 0.037 NL–UK: 0.018 HR–UK: 0.458 c,d What are the likely benefits in using an app of the \national medicines agency[ to receive safety information? It will keep me up-to-date 318 (84) 159 (83) 45 (80) 68 (88) 0.429 Increased knowledge 290 (76) 155 (81) 35 (63) 58 (75) 0.014 HR–NL: 0.004 NL–UK: 0.111 HR–UK: 0.285 Check whether symptom has been reported as ADR 248 (65) 125 (65) 33 (59) 55 (71) 0.322 Possibility to receive notifications 242 (64) 112 (59) 32 (57) 58 (75) 0.026 HR–NL: 0.842 NL–UK: 0.027 HR–UK: 0.010 Select medicine of interest 180 (47) 84 (44) 19 (34) 42 (55) 0.058 Type of news of interest e,f What type of news about medicines would be useful to you in an app? Newly identified drug–drug interactions 314 (82) 166 (87) 40 (68) 62 (79) 0.003 HR–NL: 0.001 NL–UK: 0.120 HR–UK: 0.124 New indications of a drug 286 (75) 165 (86) 28 (47) 51 (65) \ 0.001 HR–NL:\ 0.001 NL–UK: 0.035 HR–UK:\ 0.001 NCA communications 273 (71) 139 (73) 32 (54) 60 (77) 0.009 HR–NL: 0.007 NL–UK: 0.005 HR–UK: 0.482 Drugs that are taken off the market 262 (68) 126 (66) 37 (63) 58 (74) 0.288 Changes in the PIL 247 (64) 130 (68) 27 (46) 54 (69) 0.004 HR–NL: 0.002 NL–UK: 0.006 HR–UK: 0.852 704 S. T. Vries et al. Table 2 continued Total Croatia The Netherlands UK P-value DHPCs 245 (64) 135 (71) 34 (58) 42 (54) 0.016 HR–NL: 0.061 NL–UK: 0.659 HR–UK: 0.008 Educational materials 186 (49) 106 (56) 17 (29) 32 (41) 0.001 HR–NL:\ 0.001 NL–UK: 0.140 HR–UK: 0.031 Whether re-assessment is ongoing 162 (42) 74 (39) 16 (27) 41 (53) 0.009 HR–NL: 0.104 NL–UK: 0.003 HR–UK: 0.038 Discontinuation of black triangle 149 (39) 77 (40) 9 (15) 46 (59) \ 0.001 HR–NL:\ 0.001 NL–UK:\ 0.001 HR–UK: 0.005 g,h For which medicines would you like to receive news? All approved marketed drugs 137 (37) 61 (32) 20 (35) 33 (45) 0.001 All drugs I am interested in 72 (19) 44 (23) 8 (14) 9 (12) HR–NL: 0.001 Drugs related to my work 98 (26) 60 (32) 10 (18) 15 (20) NL–UK: 0.534 Drugs that I prescribe 66 (18) 23 (12) 19 (33) 17 (23) HR–UK: 0.007 Interest in other functions in the app Please think about an app that you can use for both reporting ADRs and receiving safety information. Which other information functions would f,i you like in such an app? Interactions between drugs 303 (76) 176 (92) 34 (56) 50 (61) \ 0.001 HR–NL:\ 0.001 NL–UK: 0.529 HR–UK:\ 0.001 How to resolve an ADR 296 (75) 148 (77) 42 (69) 57 (70) 0.270 Alternative drugs to the one causing the ADR 282 (71) 139 (72) 43 (70) 54 (66) 0.554 Drug product information 269 (68) 144 (75) 30 (49) 53 (65) 0.001 HR–NL:\ 0.001 NL–UK: 0.064 HR–UK: 0.080 Overview of previously reported ADRs 251 (63) 110 (57) 43 (70) 52 (63) 0.163 Prediction model 218 (55) 99 (52) 30 (49) 49 (60) 0.365 Quality alerts 154 (39) 72 (38) 15 (25) 41 (50) 0.008 HR–NL: 0.064 NL–UK: 0.002 HR–UK: 0.054 f,j Which other reporting functions would you like in such an app? When the medicine cannot be dispensed 231 (68) 124 (72) 29 (64) 46 (66) 0.460 Medicine defects 216 (64) 105 (61) 23 (51) 50 (71) 0.083 Medication errors 205 (60) 106 (62) 18 (40) 44 (63) 0.023 HR–NL: 0.009 NL–UK: 0.016 HR–UK: 0.858 Interest in an App for Two-Way Risk Communication 705 Table 2 continued Total Croatia The Netherlands UK P-value Protection of the app k,l How should an app for you to report ADRs and receive safety information of medicines be protected? Entering an email address and password 109 (30) 63 (34) 11 (20) 21 (29) 0.152 Automatic login 257 (70) 122 (66) 43 (80) 51 (71) Intention to download the app How likely are you to download a free, limited space-taking app to report ADRs and receive safety information of medicines on your device? Not at all likely 9 (2) 1 (1) 1 (2) 6 (7) \ 0.001 Slightly likely 32 (8) 6 (3) 13 (21) 8 (10) HR–NL:\ 0.001 Moderately likely 82 (21) 35 (18) 15 (24) 19 (23) NL–UK: 0.122 Very likely 276 (69) 150 (78) 33 (53) 50 (60) HR–UK:\ 0.001 Data are given as n (%) ADRs adverse drug reactions, DHPCs direct healthcare professional communications, HR Croatia, NCA national competent authority, NL Netherlands, PIL patient information leaflet All European responders The number of responders that do not want to use an app to report ADRs was 12 and the number of responders that selected ‘none’ was 7. 19 responders selected ‘Other’ Percentages are calculated excluding the responders who selected ‘None’ and those who selected ‘I do not want to use an app to report ADRs/ receive safety information’ 6 responders selected ‘None’, 13 responders selected ‘I do not want to use an app to receive safety information’ and 6 selected ‘Other’ 16 responders indicated that they did not want to receive safety information through an app and 10 selected ‘Other’ Percentages are calculated excluding those who selected ‘No other information/reporting functions’ or ‘I do not want to receive safety information through an app’ 15 responders indicated ‘None’ and 11 responders selected ‘Other’ Percentages are calculated excluding those who selected ‘None’ and ‘Other’ 2 responders were not interested in any other type of information function and 37 selected ‘Other’ The number of responders not interested in any other type of reporting function was 59 and 20 selected ‘Other’ One responder did not answer this question and 32 responders selected ‘Other’ Percentages are calculated excluding the responders who selected ‘Other’ 3.3.4 Protection of the App HCPs from The Netherlands and The UK (Table 2). HCPs from The Netherlands generally had a more negative view. Most of the HCPs preferred to use an app for two-way risk HCPs from The UK were more positive to receive news in communication via an automatic login after entering their the app about discontinuation of a black triangle for a drug e-mail address and password once (70%) (Table 2). (59 vs. 40% in Croatia and 15% in The Netherlands; Although most patients also prefer an automatic login, this overall P\0.001) and to receive quality alerts (50 vs. 38% preference was less pronounced (57%) (Table 3). in Croatia and 25% in The Netherlands; overall P = 0.008). 3.3.5 Intention to Download the App Patients from The UK had more positive views on various benefits of using the app than patients from Croatia In total, 69% of the HCPs and 52% of the patients indicated and The Netherlands (Table 3). Patients from The that it is very likely that they will download the app. Only Netherlands were more negative, particularly regarding 2% of the HCPs (Table 2) and 6% of the patients (Table 3) potential benefits of continuing an unfinished report at a indicated that this is not likely at all. later moment (33 vs. 48% in Croatia and 56% in The UK; overall P = 0.001) and of not having to contact a HCP for 3.3.6 Countries in Which the App was Already Available every symptom they experience (24 vs. 50% in Croatia and 53% in The UK; overall P\0.001). In addition, they were HCPs from Croatia appeared to have more positive views less positive about patient information leaflets (61 vs. 76% on potential benefits of the app, the addition of other in both Croatia and The UK; overall P = 0.004), infor- functionalities and the intention to download the app than mation on where to get help (36 vs. 63% in Croatia and 706 S. T. Vries et al. Table 3 Patients’ preferences and perceptions towards various characteristics of the app Total Croatia The Netherlands UK P-value Perceived benefits in using the app b,c What are the likely benefits for you in using this app? Faster way to report 513 (85) 111 (85) 138 (85) 76 (81) 0.592 Easier access to report form 434 (72) 94 (72) 103 (64) 75 (80) 0.020 HR–NL: 0.114 NL–UK: 0.007 HR–UK: 0.199 Store previous reports 290 (48) 63 (48) 70 (43) 46 (49) 0.569 Upload a photo 288 (47) 66 (51) 63 (39) 51 (54) 0.030 HR–NL: 0.042 NL–UK: 0.017 HR–UK: 0.606 Continue a report at a later moment 264 (43) 62 (48) 53 (33) 53 (56) 0.001 HR–NL: 0.009 NL–UK:\ 0.001 HR–UK: 0.199 Complete report offline and send it later 262 (43) 53 (41) 54 (33) 50 (53) 0.008 HR–NL: 0.190 NL–UK: 0.002 HR–UK: 0.066 c,d What are the likely benefits for you in using an app of the\national medicines agency[to receive safety information? Check whether symptom has been reported as ADR 441 (72) 89 (69) 106 (65) 73 (79) 0.048 HR–NL: 0.433 NL–UK: 0.014 HR–UK: 0.086 Increased knowledge 388 (63) 84 (65) 90 (55) 62 (67) 0.079 It will keep me up-to-date 378 (62) 60 (47) 112 (68) 66 (72) \ 0.001 HR–NL:\ 0.001 NL–UK: 0.565 HR–UK:\ 0.001 Possibility to receive notifications 341 (56) 42 (33) 84 (51) 62 (67) \ 0.001 HR–NL: 0.001 NL–UK: 0.012 HR–UK:\ 0.001 Select medicines of interest 311 (51) 51 (40) 70 (43) 52 (57) 0.033 HR–NL: 0.587 NL–UK: 0.033 HR–UK: 0.013 Increased confidence when talking to my HCP 283 (46) 50 (39) 75 (46) 53 (58) 0.021 HR–NL: 0.231 NL–UK: 0.068 HR–UK: 0.006 No need to contact HCP for every symptom 247 (40) 65 (50) 40 (24) 49 (53) \ 0.001 HR–NL:\ 0.001 NL–UK:\ 0.001 HR–UK: 0.674 Type of news of interest e,j What type of news about medicines would be useful to you in an app? Interest in an App for Two-Way Risk Communication 707 Table 3 continued Total Croatia The Netherlands UK P-value Safety updates 520 (84) 104 (78) 139 (84) 77 (85) 0.317 Newly identified drug interactions 439 (71) 85 (64) 105 (64) 79 (87) \0.001 HR–NL: 0.961 NL–UK:\ 0.001 HR–UK: 2009\ 0.001 Changes in the PIL 421 (68) 84 (63) 100 (61) 71 (78) 0.015 HR–NL: 0.652 NL–UK: 0.005 HR–UK: 0.018 New approved used of a drug 324 (53) 69 (52) 69 (42) 43 (47) 0.221 Whether drug review is ongoing 310 (50) 57 (43) 64 (39) 55 (60) 0.003 HR–NL: 0.477 NL–UK: 0.001 HR–UK: 0.010 News on how to take/store the drug 305 (49) 72 (54) 81 (49) 33 (36) 0.029 HR–NL: 0.386 NL–UK: 0.048 HR–UK: 0.008 Experiences of other users of the drug 292 (47) 75 (56) 79 (48) 33 (36) 0.012 HR–NL: 0.144 NL–UK: 0.073 HR–UK: 0.003 Drugs that are temporarily out of stock 243 (39) 45 (34) 65 (39) 28 (31) 0.342 g,h For which medicines would you like to receive news? All approved marketed drugs 36 (6) 10 (8) 8 (5) 7 (8) \0.001 All drugs I am interested in 161 (27) 52 (40) 20 (12) 20 (22) HR–NL:\ 0.001 All drugs to treat my disease 167 (28) 35 (27) 42 (26) 17 (19) NL–UK: 0.112 Drugs prescribed to me 241 (40) 33 (25) 93 (57) 46 (51) HR–UK: 0.001 Interest in other functions in the app Please think about an app that you can use for both reporting side effects and receiving safety information. Which other functions would you i,j like in such an app? Overview of ADRs previously reported 458 (73) 94 (70) 116 (67) 65 (68) 0.846 PIL 450 (72) 102 (76) 105 (61) 72 (76) 0.004 HR–NL: 0.004 NL–UK: 0.013 HR–UK: 0.954 Store list of medicines 405 (64) 71 (53) 102 (59) 71 (75) 0.003 HR–NL: 0.295 NL–UK: 0.010 HR–UK: 0.001 Information on where to get help 368 (59) 84 (63) 63 (36) 56 (59) \0.001 HR–NL:\ 0.001 NL–UK:\ 0.001 HR–UK: 0.567 Reminder to take medicines 321 (51) 79 (59) 54 (31) 54 (57) \0.001 HR–NL:\ 0.001 NL–UK:\ 0.001 HR–UK: 0.750 Option to report medicine defects 321 (51) 69 (51) 83 (48) 53 (56) 0.468 708 S. T. Vries et al. Table 3 continued Total Croatia The Netherlands UK P-value Learn from other patient experiences 299 (48) 77 (57) 73 (42) 50 (53) 0.024 HR–NL: 0.008 NL–UK: 0.101 HR–UK: 0.469 Chat with others about own experiences 140 (22) 41 (31) 24 (14) 26 (27) 0.001 HR–NL:\ 0.001 NL–UK: 0.007 HR–UK: 0.597 Protection of the app f,k How should an app for you to report side effects and receive safety information of medicines be protected? Entering an email address and password 255 (43) 50 (39) 58 (38) 40 (44) 0.606 Automatic login 332 (57) 79 (61) 96 (62) 51 (56) Intention to download the app How likely are you to download a free, limited space taking app to report side-effects and receive safety information of medicines on your device? Not at all likely 37 (6) 4 (3) 19 (10) 7 (7) 0.001 Slightly likely 87 (13) 11 (8) 41 (22) 17 (17) HR–NL:\ 0.001 Moderately likely 189 (29) 54 (40) 44 (24) 27 (27) NL–UK: 0.572 Very likely 341 (52) 67 (49) 83 (44) 48 (48) HR–UK: 0.035 Data are given as n (%) ADRs adverse drug reactions, DHPCs direct healthcare professional communications, HR Croatia, NCA national competent authority, NL Netherlands, PIL patient information leaflet All European responders The number of responders who do not want to use an app to report ADRs was 22, the number of responders that selected ‘none’ was 27 and 36 selected ‘Other’ Percentages are calculated excluding the responders who selected ‘None’ and those who selected ‘I do not want to use an app to report side effects/receive safety information’ 24 responders selected ‘None’, 20 responders selected ‘I do not want to use an app to receive safety information’ and 15 selected ‘Other’ 39 responders indicated that they do not want to receive safety information through an app and 29 selected ‘Other’ Percentages are calculated excluding the responders who selected ‘Other’ 27 responders indicated ‘Not applicable’ and 24 responders selected ‘Other’ Percentages are calculated excluding those who selected ‘Not applicable’ and ‘Other’ 1 responder did not complete this question, 26 were not interested in any other function and 64 selected ‘Other’ Percentages are calculated excluding those who selected ‘None/No other functions/I do not want to receive safety information through an app’ 4 responders did not answer this question and 65 responders selected ‘Other’ 2 responders did not complete this question 59% in The UK; overall P\0.001), a reminder to take characteristics and being interested in the app (Electronic medicines (31 vs. 59% in Croatia and 57% in The UK; Supplementary Material 4). overall P\0.001), and a functionality to chat with others about their own experiences (14 vs. 31% in Croatia and 3.4.1 HCP Characteristics 27% in The UK; overall P = 0.001). Of the four determinants included in the analyses of the 3.4 Determinants: HCP/Patient Characteristics HCPs, only the use of a health app was significantly associated with interest in the app. HCPs who at least Of the 399 HCPs and 656 patients who completed the sometimes use a health app were more often very interested survey, 390 and 636, respectively, were included in the than those who never use such an app (odds ratio [OR] analyses to assess the association between HCP/patient 3.52; 95% confidence interval [CI] 1.96–6.30) (Fig. 2a). The sensitivity analyses per country of interest showed that Interest in an App for Two-Way Risk Communication 709 this HCP characteristic was statistically significant for The patients were less often very interested in the app than UK only (OR 9.50; 95% CI 3.11–29.05) (Electronic Sup- younger patients (OR 0.98; 95% CI 0.97–0.997). Patients plementary Material 5). The generalised ordered logit who at least sometimes use a health app were more often model showed a similar influence of the use of health apps very interested than those who never use a health app (OR on the different levels of the outcome measure (Electronic 1.64; 95% CI 1.19–2.27) (Fig. 2b). The country-specific Supplementary Material 6). analyses showed a statistically significant association of the use of health apps for The Netherlands only (OR 2.20; 95% 3.4.2 Patient Characteristics CI 1.13–4.27) (Electronic Supplementary Material 7). Additional statistically significant associations were shown Age and use of health apps were the patient characteristics for The UK, where patients with a tertiary education level significantly associated with interest in the app. Older were less often very interested than patients with a low or N = 390 Determinants Odd raos (95% CIs) Age 1.00 (0.98-1.02) Female vs. Male 1.20 (0.76-1.89) At least sometimes use At least sometimes use of health app vs. Never 3.52 (1.96-6.30) of health apps vs. Never ADR ADR reported to agency reported to agency vs. Not 1.52 (0.94-2.47) vs. Not 0,01 0,1 1 10 100 Odds raos with 95% confidence intervals (log scale) a Healthcare professionals N = 636 Determinants Odd raos (95% CIs) Age 0.98 (0.97-0.997) Female vs. Male 1.14 (0.81-1.62) Tertiary education vs. Tertiary education vs. Low/secondary 0.75 (0.53-1.05) Low/secondary 0 medicines vs. 1-4 0.69 (0.43-1.09) ≥5 medicines vs. 1-4 1.02 (0.70-1.48) At least sometimes use 1.64 (1.19-2.27) At least sometimes use of health apps vs. Never of health apps vs. Never ADR experienced vs. Not 1.47 (0.97-2.21) Aware of reporting to Aware of reporting to agency vs. Not 1.17 (0.84-1.62) agency vs. Not 0,01 0,1 1 10 100 Odds raos with 95% confidence intervals (log scale) b Paents Fig. 2 Odds ratios with 95% confidence intervals of associations characteristics and being very interested in this app. ADR adverse between a healthcare professional characteristics and being very drug reaction, CI confidence interval interested in an app for two-way risk communication and b patient 710 S. T. Vries et al. secondary education level (OR 0.20; 95% CI 0.05–0.81), study showed that one-third of patients did not discuss their and patients who take no medicines were less often very medication symptoms with an HCP [23]. Almost half of the interested than patients who take one to four medicines patients in our study indicated that the app could increase (OR 0.17; 95% CI 0.04–0.75). The generalised ordered their confidence when talking to their HCP. This increased logit model showed a similar effect of use of health apps on certainty about an ADR may improve the patient–HCP the different levels of the outcome measure but showed conversation about ADRs. In addition, it may increase different patterns for other characteristics (Electronic patient reporting of ADRs to the national pharmacovigi- Supplementary Material 6). lance centres, but future studies will be needed to investi- gate such effects. HCPs’ and patients’ preferences and perceptions 4 Discussion towards the characteristics of the app in general were rel- atively similar. This suggests that the functionality of the This study showed that HCPs and patients were generally app can be similar for HCPs and patients. However, the interested in an app for two-way risk communication and type of drugs for which responders would like to receive that, in particular, HCPs and patients who already use a safety information differed between HCPs and patients, as health app are more interested in such an app. A main HCPs liked the option to receive information for all drugs benefit for HCPs and patients of the reporting functionality more. It should be possible to incorporate such user- in the app was that it can make the ADR reporting process specific preferences into the app. In addition, differences in faster and easier. Previous studies have shown that lack of other aspects of the app, such as appropriate terminology time and difficulty in accessing reporting forms are the for these target groups, need to be considered [2]. main barriers for HCPs to spontaneously report ADRs to Our finding that HCPs and patients who already use a national pharmacovigilance centres [12–14]. In contrast, health app are particularly interested in an app for two-way literature suggests that patients may be willing to spend risk communication suggests that, in particular, those HCPs more time on reporting an ADR than HCPs [2] and one of and patients may well be the most receptive group that the reasons for patients to spontaneously report an ADR is should be informed about the existence of such an app. when they have the impression that HCPs have too limited This could, for instance, be done via advertisements or a time to accurately report ADRs [15]. However, in this link to the app in other health apps. The high number of study patients also preferred a reporting tool that is easy responders that liked other functionalities in an app for and fast to complete. two-way risk communication also suggests that links to With respect to the information functionality of the app, various other health apps may increase its usefulness. Ways to stimulate the interest of non-app users and encouraging HCPs would like the app to keep them up-to-date with the latest news and that it will increase their knowledge about their participation could benefit from further investigation. drug safety. In addition, most HCPs liked to receive news about (newly identified) drug–drug interactions. Previ- 4.1 Strengths and Limitations ously, it has been shown that HCPs’ awareness of drug safety issues, for instance those communicated through A strength of this study is the assessment of interest in the direct healthcare professional communications (DHPCs), app among both HCPs and patients. In addition, this is a are suboptimal [16] as is their knowledge about drug–drug first study assessing the role of user characteristics on the interactions [17]. Our study suggests that HCPs are aware use of an app for two-way risk communication. Although of their lack of knowledge and that an app could be a tool we collected data from a large sample of HCPs and to improve this. patients, a limitation is that the number of responders per Patients particularly liked an overview of ADRs previ- country was still relatively low. Another limitation of this ously reported by others, and patient information leaflets as study was its methodology of a cross-sectional survey. We information in an app. They indicated that a main benefit of cannot be sure how representative it was of the studied the app would be to allow them to check whether a countries. We present data for a subgroup of three countries symptom was previously reported as an ADR. These in which the app was already available, but differences findings are in line with previous studies showing that between countries should be interpreted cautiously since patients are sometimes uncertain about an association the characteristics of the responders differ across these between a symptom and a drug [18–21], and may be countries. Moreover, the number of responders from uncertain about the exact drug causing the symptoms [22]. countries other than the three countries in which the app Providing such information in an app could reduce was available was low. Therefore, it cannot be assumed patients’ uncertainty in confirming that their symptoms are that the included population is a representative sample of caused by the drug(s) they are taking. Also, a previous the European populations. Also, we do not have any Interest in an App for Two-Way Risk Communication 711 Seventh Framework Programme FP7/2007-2013 and EFPIA compa- numbers relating to response rates since HCPs and patients nies’ in kind contribution, http://www.imi.europa.eu, UK. were reached via various channels including advertise- ments and announcements. Furthermore, survey-answering Conflict of interest Sieta T. de Vries, Petra Denig, Carmen Lasheras tendencies may differ across countries, as has been shown Ruiz, Franc ¸ois Houy ¨ ez, Lisa Wong, and Alastair Sutcliffe have no conflicts of interest that are directly relevant to the content of this previously [24]. Another limitation relates to the assess- study. Peter G. M. Mol is an employee of the Dutch Medicines ment of interest in the app. First, we could not use a val- Evaluation Board. idated measure to assess interest in the app since, to our knowledge, such a measure is not available. Second, Open Access This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International interest is a first step for actually downloading and using License (http://creativecommons.org/licenses/by-nc/4.0/), which per- the app but its actual use may be influenced by other factors mits any noncommercial use, distribution, and reproduction in any [3]. Therefore, future studies are needed to evaluate the medium, provided you give appropriate credit to the original actual use of the app in different countries. author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. 5 Conclusions References HCPs and patients in Europe are generally interested in an app for two-way risk communication, which supports its 1. Ghosh R, Lewis D. Aims and approaches of Web-RADR: a further development. Such an app should support easy and consortium ensuring reliable ADR reporting via mobile devices fast reporting of ADRs and provide information about and new insights from social media. Expert Opin Drug Saf. 2015;14(12):1845–53. drug–drug interactions and previously reported ADRs to its 2. de Vries ST, Wong L, Sutcliffe A, Houyez F, Ruiz CL, Mol PG, users. HCPs and patients who already use other health apps et al. Factors influencing the use of a mobile app for reporting are particularly interested in the app. Therefore, dissemi- adverse drug reactions and receiving safety information: a qual- nation strategies could focus on reaching these HCPs and itative study. Drug Saf. 2017;40(5):443–55. 3. Venkatesh V, Morris MG, Gordon B, Davis FD. User acceptance patients. of information technology: toward a unified view. MIS Q. Acknowledgements The healthcare professionals and patients who 2003;27(3):425–78. responded to the survey are greatly appreciated. We acknowledge 4. Ali EE, Leow JL, Chew L, Yap KY. Patients’ perception of app- other Web-RADR Work Package 3b members: Raphael van Eemeren, based educational and behavioural interventions for enhancing Karin Hace, Sandra Fernandes, Faiza Afzal and Denis Costello. oral anticancer medication adherence. J Cancer Educ. https://doi. org/10.1007/s13187-017-1248-x (Epub 2017 Jul 14). Author Contributions All authors contributed to the development 5. Lucero RJ, Frimpong JA, Fehlberg EA, Bjarnadottir RI, Weaver and formulation of the research question. All authors were involved in MT, Cook C, et al. The relationship between individual charac- distribution of the survey. STdV, PD, and PGMM conducted the teristics and interest in using a mobile phone app for HIV self- analyses. All authors contributed to the interpretation of the data. management: observational cohort study of people living with STdV wrote the manuscript. PD, CLR, FH, LW, AS, and PGMM HIV. JMIR Mhealth Uhealth. 2017;5(7):e100. reviewed and edited the manuscript. All authors have read and 6. Bender MS, Choi J, Arai S, Paul SM, Gonzalez P, Fukuoka Y. Digital technology ownership, usage, and factors predicting approved the final manuscript. downloading health apps among Caucasian, Filipino, Korean, and Latino Americans: the digital link to health survey. JMIR Compliance with Ethical Standards Mhealth Uhealth. 2014;2(4):e43. 7. Carroll JK, Moorhead A, Bond R, LeBlanc WG, Petrella RJ, Ethical approval The study protocol was submitted to (a member of) Fiscella K. Who uses mobile phone health apps and does use an ethics committee in The UK, The Netherlands and Germany. In matter? A secondary data analytics approach. J Med Internet Res. The UK, approval was obtained from the UCL Research Ethics 2017;19(4):e125. Committee (Project ID number 6855/001). In The Netherlands, the 8. Krebs P, Duncan DT. Health app use among US mobile phone Medical Ethics Committee of the University Medical Center owners: a national survey. JMIR Mhealth Uhealth. Groningen (METc UMCG) determined that ethical approval was not 2015;3(4):e101. needed for this study (reference number M16.191043). In Germany, a 9. Montastruc F, Bagheri H, Lacroix I, Damase-Michel C, Chebane review by an ethics committee was not necessary. Therefore, no L, Rousseau V, et al. Adverse drug reaction reports received further approval was deemed necessary for the other countries. All through the mobile app, VigiBIP: a comparison with classical procedures performed in studies involving human participants were in methods of reporting. Drug Saf. https://doi.org/10.1007/s40264- accordance with the ethical standards of the institutional and/or 017-0630-2 (Epub 2017 Dec 21). national research committee and with the 1964 Helsinki Declaration 10. Williams R. Generalized ordered logit/partial proportional odds and its later amendments or comparable ethical standards. models for ordinal dependent variables. Stata J. 2006;6(1):58–82. 11. Williams R. Understanding and interpreting generalized ordered Funding The Web-RADR project has received support from the logit models. J Math Sociol. 2016;40(1):7–20. Innovative Medicine Initiative Joint Undertaking (http://www.imi. 12. Vallano A, Cereza G, Pedros C, Agusti A, Danes I, Aguilera C, europa.eu) under Grant Agreement n Æ 115632, resources of which are et al. Obstacles and solutions for spontaneous reporting of composed of financial contribution from the European Union’s 712 S. T. Vries et al. adverse drug reactions in the hospital. Br J Clin Pharmacol. and assess symptoms as adverse drug reactions. Eur J Clin 2005;60(6):653–8. Pharmacol. 2014;70(5):607–15. 13. Lopez-Gonzalez E, Herdeiro MT, Figueiras A. Determinants of 19. de Vries ST, Mol PG, de Zeeuw D, Haaijer-Ruskamp FM, Denig under-reporting of adverse drug reactions: a systematic review. P. Development and initial validation of a patient-reported Drug Saf. 2009;32(1):19–31. adverse drug event questionnaire. Drug Saf. 2013;36(9):765–77. 14. De Angelis A, Colaceci S, Giusti A, Vellone E, Alvaro R. Factors 20. de Vries ST, Haaijer-Ruskamp FM, de Zeeuw D, Denig P. The that condition the spontaneous reporting of adverse drug reactions validity of a patient-reported adverse drug event questionnaire among nurses: an integrative review. J Nurs Manag. using different recall periods. Qual Life Res. 2016;24(2):151–63. 2014;23(9):2439–45. 15. Al Dweik R, Stacey D, Kohen D, Yaya S. Factors affecting 21. Britten N. Medication errors: the role of the patient. Br J Clin patient reporting of adverse drug reactions: a systematic review. Pharmacol. 2009;67(6):646–50. Br J Clin Pharmacol. 2017;83(4):875–83. 22. de Vries ST, Haaijer-Ruskamp FM, de Zeeuw D, Denig P. 16. Piening S, Haaijer-Ruskamp FM, de Graeff PA, Straus SM, Mol Construct and concurrent validity of a patient-reported adverse PG. Healthcare professionals’ self-reported experiences and drug event questionnaire: a cross-sectional study. Health Qual preferences related to direct healthcare professional communi- Life Outcomes. 2014;12:103. cations: a survey conducted in the Netherlands. Drug Saf. 23. Weingart SN, Gandhi TK, Seger AC, Seger DL, Borus J, Burdick 2012;35(11):1061–72. E, et al. Patient-reported medication symptoms in primary care. 17. Glassman PA, Simon B, Belperio P, Lanto A. Improving recog- Arch Intern Med. 2005;165(2):234–40. nition of drug interactions: benefits and barriers to using auto- 24. Tellis GJ, Chandrasekaran D. Does culture matter? Assessing mated drug alerts. Med Care. 2002;40(12):1161–71. response biases in cross-national survey research. Int J Res Mark 18. Chaipichit N, Krska J, Pratipanawatr T, Uchaipichat V, Jarern- 2010. https://ssrn.com/abstract=1659911. Accessed 30 Jan 2018. siripornkul N. A qualitative study to explore how patients identify

Journal

Drug Safety – Springer Journals

Published: Mar 2, 2018

There are no references for this article.