Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Selection biases in technology-based intervention research: patients’ technology use relates to both demographic and health-related inequities

Selection biases in technology-based intervention research: patients’ technology use relates to... Abstract Objective Researchers conduct studies with selection biases, which may limit generalizability and outcomes of intervention research. In this methodological reflection, we examined demographic and health characteristics of implantable cardioverter defibrillator patients who were excluded from an informatics intervention due to lack of access to a computer and/or the internet. Materials and Methods Using information gathered from surveys and electronic health records, we compared the intervention group to excluded patients on demographic factors, computer skills, patient activation, and medical history. Results Excluded patients were older, less educated, less engaged and activated in their health, and had worse health (ie, more medical comorbidities) than nonexcluded patients. Discussion Although excluded from the intervention based solely on lack of access to a computer and/or internet, excluded patients may have needed the intervention more because they were sicker with more comorbidities. Conclusion Researchers must be mindful of enrollment biases and demographic and health-related inequities that may exist during recruitment for technology-based interventions. research inequity, methodology, informatics, technology use, health characteristics BACKGROUND AND SIGNIFICANCE The integration of technology into health care, especially in the self-management of health, has had a profound impact on care from medical processes to health outcomes to the lived experiences of those involved—including patients, families, and clinicians.1 Methodologically sound intervention studies are necessary to determine the extent to which technologies or protocols are cost-effective, useful, and beneficial. Unfortunately, patients from disadvantaged groups (eg, seniors, minorities, and those with lower socioeconomic status [SES]) may not be using the technologies needed to test the interventions2–4 or may be using technology in ways that do not fit traditional models of use,5,6 thereby potentially creating a problem of inequitable access to study enrollment.4 This enrollment bias is just 1 research inequity that may plague informatics or other technology-related health research,7 which could potentially limit the generalizability of findings and even intervention outcomes. Although technology ownership is increasing across all demographics, those in disadvantaged groups may have limited access to websites or applications because they do not have computers or internet access.2 Notably, among those with incomes lower than $30 000, 36% do not have a smartphone and 47% do not have broadband internet service as compared to 5% and 6% of top-income earners, respectively.2 This criterion alone, that is, a patient’s ability to access the internet via a phone or computer, may create an unintended enrollment bias. Presently, it is clear that technology use is associated with demographic inequities2–4; however, it might also be related to health inequities that may decrease the generalizability of findings from medical research studies. Objective In this methodological perspective work, we discuss enrollment disparities that emerged in our informatics intervention study.8 The intervention aspect of our study necessitated use of a web-based messaging portal available via the internet; therefore, only those who had access to computers and/or internet were included in the study (and randomly assigned to either the control or intervention group). However, this created an unintended but analyzable group—a group of patients who agreed to participate in the study but did not meet intervention inclusion criteria because they did not have “access to a computer and/or the internet”—which we will refer to as “Group C.” This article focuses on the demographic and health characteristics of Group C, the unintended research inequities that may emerge when conducting technology-focused health interventions, and the benefits and shortcomings of research that employs similar methods. MATERIALS AND METHODS Background The purpose of our original study8 was to assess the impact of delivering data from remote monitoring of implantable cardioverter defibrillators (ICDs). The impact was assessed through patient engagement, patient-provider communication, and health care utilization. We anchor our discussion on health care disparity and research methodology within this case study. Study population, setting, and recruitment Participants were identified through a cardiology outpatient clinic, part of a large health system in northeast Indiana and northwest Ohio. Specifically, the Arrhythmia Diagnostic Clinic provided a list of 590 patients with a St. Jude Medical ICD to the project research nurse. The research nurse screened the patient records to include those 18 years and older and actively participating in remote monitoring. The research nurse made phone contact with eligible patients to introduce the study and inquire about interest in study enrollment. A total of 191 patients agreed to participate. See Figure 1. Figure 1. View largeDownload slide Enrollment and allocation diagram for original study. *Not contacted n includes those who were deceased, had moved, did not have a voicemail to leave a message, or who were otherwise ineligible. **Group D n includes those for whom “decline” was noted in records, but due to a methodological oversight, declines were not always noted. Figure 1. View largeDownload slide Enrollment and allocation diagram for original study. *Not contacted n includes those who were deceased, had moved, did not have a voicemail to leave a message, or who were otherwise ineligible. **Group D n includes those for whom “decline” was noted in records, but due to a methodological oversight, declines were not always noted. So that participants could be randomly assigned to study groups, only those participants with access to a computer and/or the internet were included in the intervention. A total of 144 patients were randomly assigned to 1 of 2 groups: Group A received electronic summaries of their ICD monitoring data via MyChart (the electronic patient record portal), and Group B received paper ICD summaries via postal mail. Participants who did not have access to a computer and/or the internet received only standard of care (Group C). Randomization into all 3 study groups was not possible due to limited resources; the study budget did not permit providing computers and/or internet access. Study participants received $30 compensation. Data collection At enrollment, all participants (including Group C patients) met the research nurse to complete consent forms and baseline measures. Data collection occurred from October 2014 to August 2015. All study activities for each patient took place over 6 months; however, all measures reported here were collected via surveys or data extracted from patients’ electronic health records at baseline. Relevant measures Patient characteristics Participants completed a survey containing demographic questions including age, gender, education background, race/ethnicity, income, and residential area. Additionally, participants were asked “How would you rate your ability to use a computer?” and “How would you rate your ability to use the internet?” responding on a 5-point scale (1 = very poor, 5 = very good). Finally, patients reported date of device implantation. Patient activation and engagement The validated and licensed Patient Activation Measure (PAM-13)9 was used to evaluate patient activation and engagement. Respondents indicated their agreement with 13 statements (eg, “I understand my health problems and what causes them.”) on a 4-point Likert scale (1 = Disagree Strongly, 4 = Agree Strongly), and their raw scores were transformed into a scaled score of 0–100. PAM scores were then categorized into 4 stages of activation: 1) individual deems patient role as important, 2) individual possesses confidence and health-related knowledge required to act, 3) individual takes health-related actions, and 4) individual continues actions despite stress. Medical history Patients’ histories of major medical issues (eg, diabetes, chronic lung disease, congestive heart failure, heart attack, bypass surgery, heart failure, sleep apnea, etc.) and their New York Heart Association functional class3 (Class I: No limitations for physical activity, Class II: Slight limitations on physical activity, Class III: Marked limitations for physical activity, Class IV: Physical activity causes discomfort, patient experiences heart failure symptoms at rest) were included in the ICD registry and supplemented by manual health record extractions. Analysis plan To measure the extent to which Group C patients differed from those included in the intervention, Groups A and B were combined into 1 intervention group and compared to Group C on patient characteristics, activation and engagement, and medical history. Tests for independence between the intervention group and Group C for categorical variables were conducted using chi-square tests when cell sizes were large and Fisher’s exact tests when cell sizes were small (5 or fewer). For continuous variables, independent t tests were used to test for group differences. All analyses were conducted using SAS 9.4 software. RESULTS Participants were mostly male (67.1%), white (93%), age 66 or older (58%), with a high school diploma or some college (84%), and an annual income between $20 000 and $80 000 (73%). However, compared to the intervention group, Group C patients were likely to be older, less educated, and from a lower SES (see Table 1). Additionally, Group C participants reported lesser abilities in using computers and the internet than the intervention group participants. (See Figure 2 and Supplemental Table 1). To evaluate possible confounds between computing and internet skills and income, age, health status, and education, further comparisons were conducted using independent t tests. Using these comparisons, we tested, for example, whether low-income Group C participants reported lower computing skills than their low-income intervention group counterparts. For all subsample comparisons, Group C participants had significantly lower computing and internet abilities (all Ps < .0001). See Figure 2. Figure 2. View largeDownload slide Subsample comparisons of computer and Internet abilities for Group C and Intervention group participants who had incomes < $39,999, education levels or high school or less, ages > 65, and NYHA health status classifications of Class III or IV. Figure 2. View largeDownload slide Subsample comparisons of computer and Internet abilities for Group C and Intervention group participants who had incomes < $39,999, education levels or high school or less, ages > 65, and NYHA health status classifications of Class III or IV. Table 1. Summary of patient characteristics by group and tests for independence Group C(n = 47) Intervention(n = 144) Patient characteristic % or M(SD) % or M(SD) χ2, t, or P value Gender Male 59.6 69.4 χ2(1) = 1.6, P = .21 Age 18–25 0 2.1 .049 26–35 0 0.7 36–45 0 8.3 46–55 12.8 11.8 56–65 14.9 22.2 66–75 31.9 35.4 76–85 36.2 17.4 86–90 4.3 1.4 Race White 95.7 93.1 1.00 Other 4.3 6.3 Residential area Urban 40.4 38.2 .07 Suburban 10.6 25.7 Rural 48.9 36.1 Education Did not graduate high school 23.4 5.6 .002 High school diploma/GED 42.6 33.3 Trade/some college 21.3 35.4 College graduate 10.6 15.3 Post-graduate degree 2.1 10.4 Family income $0–19 999 40.4 9.0 .0002 $20 000–39, 999 25.5 27.1 $40 000–59 999 19.2 24.3 $60 000–79 999 6.4 16.7 $80 000–99 999 2.1 8.3 over $100 000 2.1 7.6 PAM level 1 10.6 10.5 .002 2 29.8 10.5 3 36.2 30.1 4 23.4 49.0 Computing ability 1.8 (0.8) 3.3 (1.1) t(99) = 10.3, P = .0001 Internet ability 1.7 (0.8) 3.4 (1.1) t(187) = 9.7 P = .0001 Implant duration 2.9 (1.6) 3.2 (2.0) t(184) = 0.89 Group C(n = 47) Intervention(n = 144) Patient characteristic % or M(SD) % or M(SD) χ2, t, or P value Gender Male 59.6 69.4 χ2(1) = 1.6, P = .21 Age 18–25 0 2.1 .049 26–35 0 0.7 36–45 0 8.3 46–55 12.8 11.8 56–65 14.9 22.2 66–75 31.9 35.4 76–85 36.2 17.4 86–90 4.3 1.4 Race White 95.7 93.1 1.00 Other 4.3 6.3 Residential area Urban 40.4 38.2 .07 Suburban 10.6 25.7 Rural 48.9 36.1 Education Did not graduate high school 23.4 5.6 .002 High school diploma/GED 42.6 33.3 Trade/some college 21.3 35.4 College graduate 10.6 15.3 Post-graduate degree 2.1 10.4 Family income $0–19 999 40.4 9.0 .0002 $20 000–39, 999 25.5 27.1 $40 000–59 999 19.2 24.3 $60 000–79 999 6.4 16.7 $80 000–99 999 2.1 8.3 over $100 000 2.1 7.6 PAM level 1 10.6 10.5 .002 2 29.8 10.5 3 36.2 30.1 4 23.4 49.0 Computing ability 1.8 (0.8) 3.3 (1.1) t(99) = 10.3, P = .0001 Internet ability 1.7 (0.8) 3.4 (1.1) t(187) = 9.7 P = .0001 Implant duration 2.9 (1.6) 3.2 (2.0) t(184) = 0.89 Abbreviations: GED, general education diploma; SD, standard deviation; M, Mean; PAM, Patient Activation Measure View Large Table 1. Summary of patient characteristics by group and tests for independence Group C(n = 47) Intervention(n = 144) Patient characteristic % or M(SD) % or M(SD) χ2, t, or P value Gender Male 59.6 69.4 χ2(1) = 1.6, P = .21 Age 18–25 0 2.1 .049 26–35 0 0.7 36–45 0 8.3 46–55 12.8 11.8 56–65 14.9 22.2 66–75 31.9 35.4 76–85 36.2 17.4 86–90 4.3 1.4 Race White 95.7 93.1 1.00 Other 4.3 6.3 Residential area Urban 40.4 38.2 .07 Suburban 10.6 25.7 Rural 48.9 36.1 Education Did not graduate high school 23.4 5.6 .002 High school diploma/GED 42.6 33.3 Trade/some college 21.3 35.4 College graduate 10.6 15.3 Post-graduate degree 2.1 10.4 Family income $0–19 999 40.4 9.0 .0002 $20 000–39, 999 25.5 27.1 $40 000–59 999 19.2 24.3 $60 000–79 999 6.4 16.7 $80 000–99 999 2.1 8.3 over $100 000 2.1 7.6 PAM level 1 10.6 10.5 .002 2 29.8 10.5 3 36.2 30.1 4 23.4 49.0 Computing ability 1.8 (0.8) 3.3 (1.1) t(99) = 10.3, P = .0001 Internet ability 1.7 (0.8) 3.4 (1.1) t(187) = 9.7 P = .0001 Implant duration 2.9 (1.6) 3.2 (2.0) t(184) = 0.89 Group C(n = 47) Intervention(n = 144) Patient characteristic % or M(SD) % or M(SD) χ2, t, or P value Gender Male 59.6 69.4 χ2(1) = 1.6, P = .21 Age 18–25 0 2.1 .049 26–35 0 0.7 36–45 0 8.3 46–55 12.8 11.8 56–65 14.9 22.2 66–75 31.9 35.4 76–85 36.2 17.4 86–90 4.3 1.4 Race White 95.7 93.1 1.00 Other 4.3 6.3 Residential area Urban 40.4 38.2 .07 Suburban 10.6 25.7 Rural 48.9 36.1 Education Did not graduate high school 23.4 5.6 .002 High school diploma/GED 42.6 33.3 Trade/some college 21.3 35.4 College graduate 10.6 15.3 Post-graduate degree 2.1 10.4 Family income $0–19 999 40.4 9.0 .0002 $20 000–39, 999 25.5 27.1 $40 000–59 999 19.2 24.3 $60 000–79 999 6.4 16.7 $80 000–99 999 2.1 8.3 over $100 000 2.1 7.6 PAM level 1 10.6 10.5 .002 2 29.8 10.5 3 36.2 30.1 4 23.4 49.0 Computing ability 1.8 (0.8) 3.3 (1.1) t(99) = 10.3, P = .0001 Internet ability 1.7 (0.8) 3.4 (1.1) t(187) = 9.7 P = .0001 Implant duration 2.9 (1.6) 3.2 (2.0) t(184) = 0.89 Abbreviations: GED, general education diploma; SD, standard deviation; M, Mean; PAM, Patient Activation Measure View Large With regard to patient activation, 49% of the intervention group met the score criterion for Level 4, whereas only 23.4% of Group C met the Level 4 criterion. In a test for independence, patient activation level and membership in Group C were not independent. Finally, in terms of medical history (Table 2), Group C patients were more likely to be in a higher functional class in terms of physical activity limitations (indicating greater limitations), and were more likely to have histories of congestive heart failure, coronary heart disease, heart attacks, bypass surgery, chronic lung disease, and diabetes. In terms of other medical history items we analyzed (eg, sleep apnea), there were no significant differences between groups. Table 2. Summary of medical history (percentage) by group (n = 191) and P value for test of independence Medical history Group C(n = 47) Intervention(n = 144) Test of independence % % χ2 or P value NYHA functional class Class I 7.0 26.4 .03 Class II 46.5 40.3 Class III 44.2 31.8 Class IV 2.3 1.6 Congestive heart failure 93.0 74.5 .009 Coronary heart disease 88.0 65.3 .04 Prior heart attack 51.1 33.6 χ2(1) = 4.4, P = .04 Prior bypass surgery 43.2 19.7 χ2(1) = 9.7, P = .002 Chronic lung disease 27.3 9.5 χ2(1) = 8.8, P = .003 Diabetes 44.4 21.2 χ2(1) = 9.3, P = .002 Medical history Group C(n = 47) Intervention(n = 144) Test of independence % % χ2 or P value NYHA functional class Class I 7.0 26.4 .03 Class II 46.5 40.3 Class III 44.2 31.8 Class IV 2.3 1.6 Congestive heart failure 93.0 74.5 .009 Coronary heart disease 88.0 65.3 .04 Prior heart attack 51.1 33.6 χ2(1) = 4.4, P = .04 Prior bypass surgery 43.2 19.7 χ2(1) = 9.7, P = .002 Chronic lung disease 27.3 9.5 χ2(1) = 8.8, P = .003 Diabetes 44.4 21.2 χ2(1) = 9.3, P = .002 Abbreviation: NYHA, New York Heart Association. View Large Table 2. Summary of medical history (percentage) by group (n = 191) and P value for test of independence Medical history Group C(n = 47) Intervention(n = 144) Test of independence % % χ2 or P value NYHA functional class Class I 7.0 26.4 .03 Class II 46.5 40.3 Class III 44.2 31.8 Class IV 2.3 1.6 Congestive heart failure 93.0 74.5 .009 Coronary heart disease 88.0 65.3 .04 Prior heart attack 51.1 33.6 χ2(1) = 4.4, P = .04 Prior bypass surgery 43.2 19.7 χ2(1) = 9.7, P = .002 Chronic lung disease 27.3 9.5 χ2(1) = 8.8, P = .003 Diabetes 44.4 21.2 χ2(1) = 9.3, P = .002 Medical history Group C(n = 47) Intervention(n = 144) Test of independence % % χ2 or P value NYHA functional class Class I 7.0 26.4 .03 Class II 46.5 40.3 Class III 44.2 31.8 Class IV 2.3 1.6 Congestive heart failure 93.0 74.5 .009 Coronary heart disease 88.0 65.3 .04 Prior heart attack 51.1 33.6 χ2(1) = 4.4, P = .04 Prior bypass surgery 43.2 19.7 χ2(1) = 9.7, P = .002 Chronic lung disease 27.3 9.5 χ2(1) = 8.8, P = .003 Diabetes 44.4 21.2 χ2(1) = 9.3, P = .002 Abbreviation: NYHA, New York Heart Association. View Large DISCUSSION Recently, researchers have suggested that mHealth interventions can widen inequity gaps because underprivileged populations may not have access to technologies or uptake or adhere to technological interventions.4 However, a related problem is that this lack of access can lead to enrollment inequities7 that might limit the generalizability of research findings. Previous research has shown that demographic factors (eg, older age and lower SES) are associated with a lack of access to or familiarity with technology.2–4 Our study provided additional support for those findings. However, we also found that patients with limited technology access also had a host of medical comorbidities, including a history of diabetes and chronic lung disease, among others, and were less engaged in their health care than those with access. Although computer and internet access is associated with SES, as we showed, these variables are not entirely overlapping, and even older, sicker, and lower SES patients may still report having computer and internet access and abilities. The findings from this research illustrate a need to re-examine methodological approaches, and associated tensions, within technology studies. Those who did not have access to computers and/or the internet were not only older, less educated, and from a lower SES, but they were also in poorer health than those who did have access. This caused a research inequity at enrollment such that those who were most disadvantaged in several key ways were not enrolled in either intervention group. Clearly, this limits the generalizability of our main study findings,8 as we are uncertain about whether our findings might generalize to these disadvantaged populations. However, we also note that our intervention contrasted the effectiveness of a computer-based versus a paper-based remote monitoring report for providing patients feedback about their implanted device, and we found that the paper-based intervention was just as effective.8 Thus, although we did not include patients who did not use computers in our intervention group (to satisfy random assignment), the results of the study may improve care for these very patients. CONCLUSION Informatics, clinical, and health science researchers should make efforts to include patients with limited technology access who might be most in need of health-related interventions. As technology offers opportunities for wide-based disbursement of health information and care, it remains a promising mechanism for reaching underserved populations. However, intervention efforts to include disadvantaged patients should also include provision of and training in relevant technological tools; and uptake, utilization, and challenges associated with these tools should continue to be studied.10 Future research might also examine the characteristics of “Group D” patients (ie, patients who declined participation), as we suspect that these patients might be even more disadvantaged in terms of demographic and health characteristics. As these patients are hard to reach, commercial recruitment services might be necessary for enrollment. However, even without Group C and Group D patients, it is still possible for interventions to benefit these populations (eg, through testing technology methods against nontechnology methods) as long as sample limitations are acknowledged. FUNDING This work was supported by the Office of National Coordinator for Health Information Technology, grant number EP-HIT-10-002 and a St. Jude Medical Investigator Initiated Research award. CONTRIBUTORS TT and MM helped to conceptualize, design, and conduct this study. MD, JP, MF, and RP analyzed and/or interpreted the data and wrote the manuscript with input from all authors. TT was in charge of overall direction and planning of the project. Additionally, all authors drafted or revised this manuscript critically for important intellectual content, approved the version to be published, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. ACKNOWLEDGEMENTS We thank Carly Daley and Shauna Wagner for their assistance with writing this article. We also thank all of the patients who participated in this research. CONFLICT OF INTEREST STATEMENT Dr Michael J. Mirro has research funding from the Agency for Healthcare Research and Quality and the following financial relationships with industry to disclose: research grants from Biotronik, Inc., Medtronic, Inc., and Janssen Scientific Affairs, LLC; consulting fees/honoraria with McKesson Corporation, iRhythm Technologies, Inc., and Zoll Medical Corporation; and financial partnership with Medical Informatics Engineering, outside the submitted work. Dr Michael J. Mirro’s relationships with academia include serving as trustee of Indiana University and on the Indiana University Health Board. Dr Tammy Toscos has research funding from the Agency for Healthcare Research and Quality and the following financial relationships with industry to disclose: research grants from Biotronik, Inc., Medtronic, Inc., and Janssen Scientific Affairs, LLC; and iRhythm Technologies Inc. outside the submitted work. REFERENCES 1 Banova B. The impact of technology on healthcare. American Institute of Medical Sciences and Education (AIMS); 2018 . https://www.aimseducation.edu/blog/the-impact-of-technology-on-healthcare/. Accessed January 6, 2019. 2 Anderson M. Digital Divide Persists Even as Lower-Income Americans Make Gains in Tech Adoption . Washington, DC : Pew Research Center ; 2017 . http://www.pewresearch.org/fact-tank/2017/03/22/digital-divide-persists-even-as-lower-income-americans-make-gains-in-tech-adoption/. Accessed January 6, 2019. Google Preview WorldCat COPAC 3 Ginossar T , Nelson S. Reducing the health and digital divides: a model for using community-based participatory research approach to e-health interventions in low-income Hispanic communities . J Comput-Mediated Commun 2010 ; 15 ( 4 ): 530 – 551 . Google Scholar Crossref Search ADS WorldCat 4 Veinot TC , Mitchell H , Ancker JS. Good intentions are not enough: how informatics interventions can worsen inequality . JAMIA 2018 ; 25 ( 8 ): 1080 – 1088 . Google Scholar PubMed WorldCat 5 Jacobs M , Cramer H , Barkhuus L. Caring about sharing: couples’ practices in single user device access. In: Proceedings of the 19th Internatinal Conference on Supporting Group Work . New York, NY : ACM ; 2016 : 235 – 243 . Google Preview WorldCat COPAC 6 Matthews T , Liao K , Turner A , Berkovich M , Reeder R , Consolvo S. “She’ll just grab any device that’s closer”: a study of everyday device & account sharing in households In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems . New York, NY : ACM ; 2016 : 5921 – 5932 . Google Preview WorldCat COPAC 7 López CM , Qanungo S , Jenkins CM , Acierno R. Technology as a means to address disparities in mental health research: a guide to “tele-tailoring” your research methods . Prof Psychol Res Pract 2018 ; 49 ( 1 ): 57 – 64 . Google Scholar Crossref Search ADS WorldCat 8 Mirro M , Daley C , Wagner S , Rohani Ghahari R , Drouin M , Toscos T. Delivering remote monitoring data to patients with implantable cardioverter‐defibrillators: does medium matter? Pacing Clin Electrophysiol 2018 ; 41 ( 11 ): 1526 – 1535 . Google Scholar Crossref Search ADS PubMed WorldCat 9 Hibbard JH , Mahoney ER , Stockard J , Tusler M. Development and testing of a short form of the patient activation measure . Health Serv Res 2005 ; 40 ( 6 Pt 1 ): 1918 – 1930 . Google Scholar Crossref Search ADS PubMed WorldCat 10 Anderson-Lewis C , Darville G , Mercado RE , Howell S , Di Maggio S. mHealth technology use and implications in historically underserved and minority populations in the United States: systematic literature review . JMIR Mhealth Uhealth 2018 ; 6 ( 6 ): e128 . Google Scholar Crossref Search ADS PubMed WorldCat © The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of the American Medical Informatics Association Oxford University Press

Selection biases in technology-based intervention research: patients’ technology use relates to both demographic and health-related inequities

Loading next page...
 
/lp/oxford-university-press/selection-biases-in-technology-based-intervention-research-patients-3sMFrLKTrD

References (16)

Publisher
Oxford University Press
Copyright
© The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
1067-5027
eISSN
1527-974X
DOI
10.1093/jamia/ocz058
Publisher site
See Article on Publisher Site

Abstract

Abstract Objective Researchers conduct studies with selection biases, which may limit generalizability and outcomes of intervention research. In this methodological reflection, we examined demographic and health characteristics of implantable cardioverter defibrillator patients who were excluded from an informatics intervention due to lack of access to a computer and/or the internet. Materials and Methods Using information gathered from surveys and electronic health records, we compared the intervention group to excluded patients on demographic factors, computer skills, patient activation, and medical history. Results Excluded patients were older, less educated, less engaged and activated in their health, and had worse health (ie, more medical comorbidities) than nonexcluded patients. Discussion Although excluded from the intervention based solely on lack of access to a computer and/or internet, excluded patients may have needed the intervention more because they were sicker with more comorbidities. Conclusion Researchers must be mindful of enrollment biases and demographic and health-related inequities that may exist during recruitment for technology-based interventions. research inequity, methodology, informatics, technology use, health characteristics BACKGROUND AND SIGNIFICANCE The integration of technology into health care, especially in the self-management of health, has had a profound impact on care from medical processes to health outcomes to the lived experiences of those involved—including patients, families, and clinicians.1 Methodologically sound intervention studies are necessary to determine the extent to which technologies or protocols are cost-effective, useful, and beneficial. Unfortunately, patients from disadvantaged groups (eg, seniors, minorities, and those with lower socioeconomic status [SES]) may not be using the technologies needed to test the interventions2–4 or may be using technology in ways that do not fit traditional models of use,5,6 thereby potentially creating a problem of inequitable access to study enrollment.4 This enrollment bias is just 1 research inequity that may plague informatics or other technology-related health research,7 which could potentially limit the generalizability of findings and even intervention outcomes. Although technology ownership is increasing across all demographics, those in disadvantaged groups may have limited access to websites or applications because they do not have computers or internet access.2 Notably, among those with incomes lower than $30 000, 36% do not have a smartphone and 47% do not have broadband internet service as compared to 5% and 6% of top-income earners, respectively.2 This criterion alone, that is, a patient’s ability to access the internet via a phone or computer, may create an unintended enrollment bias. Presently, it is clear that technology use is associated with demographic inequities2–4; however, it might also be related to health inequities that may decrease the generalizability of findings from medical research studies. Objective In this methodological perspective work, we discuss enrollment disparities that emerged in our informatics intervention study.8 The intervention aspect of our study necessitated use of a web-based messaging portal available via the internet; therefore, only those who had access to computers and/or internet were included in the study (and randomly assigned to either the control or intervention group). However, this created an unintended but analyzable group—a group of patients who agreed to participate in the study but did not meet intervention inclusion criteria because they did not have “access to a computer and/or the internet”—which we will refer to as “Group C.” This article focuses on the demographic and health characteristics of Group C, the unintended research inequities that may emerge when conducting technology-focused health interventions, and the benefits and shortcomings of research that employs similar methods. MATERIALS AND METHODS Background The purpose of our original study8 was to assess the impact of delivering data from remote monitoring of implantable cardioverter defibrillators (ICDs). The impact was assessed through patient engagement, patient-provider communication, and health care utilization. We anchor our discussion on health care disparity and research methodology within this case study. Study population, setting, and recruitment Participants were identified through a cardiology outpatient clinic, part of a large health system in northeast Indiana and northwest Ohio. Specifically, the Arrhythmia Diagnostic Clinic provided a list of 590 patients with a St. Jude Medical ICD to the project research nurse. The research nurse screened the patient records to include those 18 years and older and actively participating in remote monitoring. The research nurse made phone contact with eligible patients to introduce the study and inquire about interest in study enrollment. A total of 191 patients agreed to participate. See Figure 1. Figure 1. View largeDownload slide Enrollment and allocation diagram for original study. *Not contacted n includes those who were deceased, had moved, did not have a voicemail to leave a message, or who were otherwise ineligible. **Group D n includes those for whom “decline” was noted in records, but due to a methodological oversight, declines were not always noted. Figure 1. View largeDownload slide Enrollment and allocation diagram for original study. *Not contacted n includes those who were deceased, had moved, did not have a voicemail to leave a message, or who were otherwise ineligible. **Group D n includes those for whom “decline” was noted in records, but due to a methodological oversight, declines were not always noted. So that participants could be randomly assigned to study groups, only those participants with access to a computer and/or the internet were included in the intervention. A total of 144 patients were randomly assigned to 1 of 2 groups: Group A received electronic summaries of their ICD monitoring data via MyChart (the electronic patient record portal), and Group B received paper ICD summaries via postal mail. Participants who did not have access to a computer and/or the internet received only standard of care (Group C). Randomization into all 3 study groups was not possible due to limited resources; the study budget did not permit providing computers and/or internet access. Study participants received $30 compensation. Data collection At enrollment, all participants (including Group C patients) met the research nurse to complete consent forms and baseline measures. Data collection occurred from October 2014 to August 2015. All study activities for each patient took place over 6 months; however, all measures reported here were collected via surveys or data extracted from patients’ electronic health records at baseline. Relevant measures Patient characteristics Participants completed a survey containing demographic questions including age, gender, education background, race/ethnicity, income, and residential area. Additionally, participants were asked “How would you rate your ability to use a computer?” and “How would you rate your ability to use the internet?” responding on a 5-point scale (1 = very poor, 5 = very good). Finally, patients reported date of device implantation. Patient activation and engagement The validated and licensed Patient Activation Measure (PAM-13)9 was used to evaluate patient activation and engagement. Respondents indicated their agreement with 13 statements (eg, “I understand my health problems and what causes them.”) on a 4-point Likert scale (1 = Disagree Strongly, 4 = Agree Strongly), and their raw scores were transformed into a scaled score of 0–100. PAM scores were then categorized into 4 stages of activation: 1) individual deems patient role as important, 2) individual possesses confidence and health-related knowledge required to act, 3) individual takes health-related actions, and 4) individual continues actions despite stress. Medical history Patients’ histories of major medical issues (eg, diabetes, chronic lung disease, congestive heart failure, heart attack, bypass surgery, heart failure, sleep apnea, etc.) and their New York Heart Association functional class3 (Class I: No limitations for physical activity, Class II: Slight limitations on physical activity, Class III: Marked limitations for physical activity, Class IV: Physical activity causes discomfort, patient experiences heart failure symptoms at rest) were included in the ICD registry and supplemented by manual health record extractions. Analysis plan To measure the extent to which Group C patients differed from those included in the intervention, Groups A and B were combined into 1 intervention group and compared to Group C on patient characteristics, activation and engagement, and medical history. Tests for independence between the intervention group and Group C for categorical variables were conducted using chi-square tests when cell sizes were large and Fisher’s exact tests when cell sizes were small (5 or fewer). For continuous variables, independent t tests were used to test for group differences. All analyses were conducted using SAS 9.4 software. RESULTS Participants were mostly male (67.1%), white (93%), age 66 or older (58%), with a high school diploma or some college (84%), and an annual income between $20 000 and $80 000 (73%). However, compared to the intervention group, Group C patients were likely to be older, less educated, and from a lower SES (see Table 1). Additionally, Group C participants reported lesser abilities in using computers and the internet than the intervention group participants. (See Figure 2 and Supplemental Table 1). To evaluate possible confounds between computing and internet skills and income, age, health status, and education, further comparisons were conducted using independent t tests. Using these comparisons, we tested, for example, whether low-income Group C participants reported lower computing skills than their low-income intervention group counterparts. For all subsample comparisons, Group C participants had significantly lower computing and internet abilities (all Ps < .0001). See Figure 2. Figure 2. View largeDownload slide Subsample comparisons of computer and Internet abilities for Group C and Intervention group participants who had incomes < $39,999, education levels or high school or less, ages > 65, and NYHA health status classifications of Class III or IV. Figure 2. View largeDownload slide Subsample comparisons of computer and Internet abilities for Group C and Intervention group participants who had incomes < $39,999, education levels or high school or less, ages > 65, and NYHA health status classifications of Class III or IV. Table 1. Summary of patient characteristics by group and tests for independence Group C(n = 47) Intervention(n = 144) Patient characteristic % or M(SD) % or M(SD) χ2, t, or P value Gender Male 59.6 69.4 χ2(1) = 1.6, P = .21 Age 18–25 0 2.1 .049 26–35 0 0.7 36–45 0 8.3 46–55 12.8 11.8 56–65 14.9 22.2 66–75 31.9 35.4 76–85 36.2 17.4 86–90 4.3 1.4 Race White 95.7 93.1 1.00 Other 4.3 6.3 Residential area Urban 40.4 38.2 .07 Suburban 10.6 25.7 Rural 48.9 36.1 Education Did not graduate high school 23.4 5.6 .002 High school diploma/GED 42.6 33.3 Trade/some college 21.3 35.4 College graduate 10.6 15.3 Post-graduate degree 2.1 10.4 Family income $0–19 999 40.4 9.0 .0002 $20 000–39, 999 25.5 27.1 $40 000–59 999 19.2 24.3 $60 000–79 999 6.4 16.7 $80 000–99 999 2.1 8.3 over $100 000 2.1 7.6 PAM level 1 10.6 10.5 .002 2 29.8 10.5 3 36.2 30.1 4 23.4 49.0 Computing ability 1.8 (0.8) 3.3 (1.1) t(99) = 10.3, P = .0001 Internet ability 1.7 (0.8) 3.4 (1.1) t(187) = 9.7 P = .0001 Implant duration 2.9 (1.6) 3.2 (2.0) t(184) = 0.89 Group C(n = 47) Intervention(n = 144) Patient characteristic % or M(SD) % or M(SD) χ2, t, or P value Gender Male 59.6 69.4 χ2(1) = 1.6, P = .21 Age 18–25 0 2.1 .049 26–35 0 0.7 36–45 0 8.3 46–55 12.8 11.8 56–65 14.9 22.2 66–75 31.9 35.4 76–85 36.2 17.4 86–90 4.3 1.4 Race White 95.7 93.1 1.00 Other 4.3 6.3 Residential area Urban 40.4 38.2 .07 Suburban 10.6 25.7 Rural 48.9 36.1 Education Did not graduate high school 23.4 5.6 .002 High school diploma/GED 42.6 33.3 Trade/some college 21.3 35.4 College graduate 10.6 15.3 Post-graduate degree 2.1 10.4 Family income $0–19 999 40.4 9.0 .0002 $20 000–39, 999 25.5 27.1 $40 000–59 999 19.2 24.3 $60 000–79 999 6.4 16.7 $80 000–99 999 2.1 8.3 over $100 000 2.1 7.6 PAM level 1 10.6 10.5 .002 2 29.8 10.5 3 36.2 30.1 4 23.4 49.0 Computing ability 1.8 (0.8) 3.3 (1.1) t(99) = 10.3, P = .0001 Internet ability 1.7 (0.8) 3.4 (1.1) t(187) = 9.7 P = .0001 Implant duration 2.9 (1.6) 3.2 (2.0) t(184) = 0.89 Abbreviations: GED, general education diploma; SD, standard deviation; M, Mean; PAM, Patient Activation Measure View Large Table 1. Summary of patient characteristics by group and tests for independence Group C(n = 47) Intervention(n = 144) Patient characteristic % or M(SD) % or M(SD) χ2, t, or P value Gender Male 59.6 69.4 χ2(1) = 1.6, P = .21 Age 18–25 0 2.1 .049 26–35 0 0.7 36–45 0 8.3 46–55 12.8 11.8 56–65 14.9 22.2 66–75 31.9 35.4 76–85 36.2 17.4 86–90 4.3 1.4 Race White 95.7 93.1 1.00 Other 4.3 6.3 Residential area Urban 40.4 38.2 .07 Suburban 10.6 25.7 Rural 48.9 36.1 Education Did not graduate high school 23.4 5.6 .002 High school diploma/GED 42.6 33.3 Trade/some college 21.3 35.4 College graduate 10.6 15.3 Post-graduate degree 2.1 10.4 Family income $0–19 999 40.4 9.0 .0002 $20 000–39, 999 25.5 27.1 $40 000–59 999 19.2 24.3 $60 000–79 999 6.4 16.7 $80 000–99 999 2.1 8.3 over $100 000 2.1 7.6 PAM level 1 10.6 10.5 .002 2 29.8 10.5 3 36.2 30.1 4 23.4 49.0 Computing ability 1.8 (0.8) 3.3 (1.1) t(99) = 10.3, P = .0001 Internet ability 1.7 (0.8) 3.4 (1.1) t(187) = 9.7 P = .0001 Implant duration 2.9 (1.6) 3.2 (2.0) t(184) = 0.89 Group C(n = 47) Intervention(n = 144) Patient characteristic % or M(SD) % or M(SD) χ2, t, or P value Gender Male 59.6 69.4 χ2(1) = 1.6, P = .21 Age 18–25 0 2.1 .049 26–35 0 0.7 36–45 0 8.3 46–55 12.8 11.8 56–65 14.9 22.2 66–75 31.9 35.4 76–85 36.2 17.4 86–90 4.3 1.4 Race White 95.7 93.1 1.00 Other 4.3 6.3 Residential area Urban 40.4 38.2 .07 Suburban 10.6 25.7 Rural 48.9 36.1 Education Did not graduate high school 23.4 5.6 .002 High school diploma/GED 42.6 33.3 Trade/some college 21.3 35.4 College graduate 10.6 15.3 Post-graduate degree 2.1 10.4 Family income $0–19 999 40.4 9.0 .0002 $20 000–39, 999 25.5 27.1 $40 000–59 999 19.2 24.3 $60 000–79 999 6.4 16.7 $80 000–99 999 2.1 8.3 over $100 000 2.1 7.6 PAM level 1 10.6 10.5 .002 2 29.8 10.5 3 36.2 30.1 4 23.4 49.0 Computing ability 1.8 (0.8) 3.3 (1.1) t(99) = 10.3, P = .0001 Internet ability 1.7 (0.8) 3.4 (1.1) t(187) = 9.7 P = .0001 Implant duration 2.9 (1.6) 3.2 (2.0) t(184) = 0.89 Abbreviations: GED, general education diploma; SD, standard deviation; M, Mean; PAM, Patient Activation Measure View Large With regard to patient activation, 49% of the intervention group met the score criterion for Level 4, whereas only 23.4% of Group C met the Level 4 criterion. In a test for independence, patient activation level and membership in Group C were not independent. Finally, in terms of medical history (Table 2), Group C patients were more likely to be in a higher functional class in terms of physical activity limitations (indicating greater limitations), and were more likely to have histories of congestive heart failure, coronary heart disease, heart attacks, bypass surgery, chronic lung disease, and diabetes. In terms of other medical history items we analyzed (eg, sleep apnea), there were no significant differences between groups. Table 2. Summary of medical history (percentage) by group (n = 191) and P value for test of independence Medical history Group C(n = 47) Intervention(n = 144) Test of independence % % χ2 or P value NYHA functional class Class I 7.0 26.4 .03 Class II 46.5 40.3 Class III 44.2 31.8 Class IV 2.3 1.6 Congestive heart failure 93.0 74.5 .009 Coronary heart disease 88.0 65.3 .04 Prior heart attack 51.1 33.6 χ2(1) = 4.4, P = .04 Prior bypass surgery 43.2 19.7 χ2(1) = 9.7, P = .002 Chronic lung disease 27.3 9.5 χ2(1) = 8.8, P = .003 Diabetes 44.4 21.2 χ2(1) = 9.3, P = .002 Medical history Group C(n = 47) Intervention(n = 144) Test of independence % % χ2 or P value NYHA functional class Class I 7.0 26.4 .03 Class II 46.5 40.3 Class III 44.2 31.8 Class IV 2.3 1.6 Congestive heart failure 93.0 74.5 .009 Coronary heart disease 88.0 65.3 .04 Prior heart attack 51.1 33.6 χ2(1) = 4.4, P = .04 Prior bypass surgery 43.2 19.7 χ2(1) = 9.7, P = .002 Chronic lung disease 27.3 9.5 χ2(1) = 8.8, P = .003 Diabetes 44.4 21.2 χ2(1) = 9.3, P = .002 Abbreviation: NYHA, New York Heart Association. View Large Table 2. Summary of medical history (percentage) by group (n = 191) and P value for test of independence Medical history Group C(n = 47) Intervention(n = 144) Test of independence % % χ2 or P value NYHA functional class Class I 7.0 26.4 .03 Class II 46.5 40.3 Class III 44.2 31.8 Class IV 2.3 1.6 Congestive heart failure 93.0 74.5 .009 Coronary heart disease 88.0 65.3 .04 Prior heart attack 51.1 33.6 χ2(1) = 4.4, P = .04 Prior bypass surgery 43.2 19.7 χ2(1) = 9.7, P = .002 Chronic lung disease 27.3 9.5 χ2(1) = 8.8, P = .003 Diabetes 44.4 21.2 χ2(1) = 9.3, P = .002 Medical history Group C(n = 47) Intervention(n = 144) Test of independence % % χ2 or P value NYHA functional class Class I 7.0 26.4 .03 Class II 46.5 40.3 Class III 44.2 31.8 Class IV 2.3 1.6 Congestive heart failure 93.0 74.5 .009 Coronary heart disease 88.0 65.3 .04 Prior heart attack 51.1 33.6 χ2(1) = 4.4, P = .04 Prior bypass surgery 43.2 19.7 χ2(1) = 9.7, P = .002 Chronic lung disease 27.3 9.5 χ2(1) = 8.8, P = .003 Diabetes 44.4 21.2 χ2(1) = 9.3, P = .002 Abbreviation: NYHA, New York Heart Association. View Large DISCUSSION Recently, researchers have suggested that mHealth interventions can widen inequity gaps because underprivileged populations may not have access to technologies or uptake or adhere to technological interventions.4 However, a related problem is that this lack of access can lead to enrollment inequities7 that might limit the generalizability of research findings. Previous research has shown that demographic factors (eg, older age and lower SES) are associated with a lack of access to or familiarity with technology.2–4 Our study provided additional support for those findings. However, we also found that patients with limited technology access also had a host of medical comorbidities, including a history of diabetes and chronic lung disease, among others, and were less engaged in their health care than those with access. Although computer and internet access is associated with SES, as we showed, these variables are not entirely overlapping, and even older, sicker, and lower SES patients may still report having computer and internet access and abilities. The findings from this research illustrate a need to re-examine methodological approaches, and associated tensions, within technology studies. Those who did not have access to computers and/or the internet were not only older, less educated, and from a lower SES, but they were also in poorer health than those who did have access. This caused a research inequity at enrollment such that those who were most disadvantaged in several key ways were not enrolled in either intervention group. Clearly, this limits the generalizability of our main study findings,8 as we are uncertain about whether our findings might generalize to these disadvantaged populations. However, we also note that our intervention contrasted the effectiveness of a computer-based versus a paper-based remote monitoring report for providing patients feedback about their implanted device, and we found that the paper-based intervention was just as effective.8 Thus, although we did not include patients who did not use computers in our intervention group (to satisfy random assignment), the results of the study may improve care for these very patients. CONCLUSION Informatics, clinical, and health science researchers should make efforts to include patients with limited technology access who might be most in need of health-related interventions. As technology offers opportunities for wide-based disbursement of health information and care, it remains a promising mechanism for reaching underserved populations. However, intervention efforts to include disadvantaged patients should also include provision of and training in relevant technological tools; and uptake, utilization, and challenges associated with these tools should continue to be studied.10 Future research might also examine the characteristics of “Group D” patients (ie, patients who declined participation), as we suspect that these patients might be even more disadvantaged in terms of demographic and health characteristics. As these patients are hard to reach, commercial recruitment services might be necessary for enrollment. However, even without Group C and Group D patients, it is still possible for interventions to benefit these populations (eg, through testing technology methods against nontechnology methods) as long as sample limitations are acknowledged. FUNDING This work was supported by the Office of National Coordinator for Health Information Technology, grant number EP-HIT-10-002 and a St. Jude Medical Investigator Initiated Research award. CONTRIBUTORS TT and MM helped to conceptualize, design, and conduct this study. MD, JP, MF, and RP analyzed and/or interpreted the data and wrote the manuscript with input from all authors. TT was in charge of overall direction and planning of the project. Additionally, all authors drafted or revised this manuscript critically for important intellectual content, approved the version to be published, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. ACKNOWLEDGEMENTS We thank Carly Daley and Shauna Wagner for their assistance with writing this article. We also thank all of the patients who participated in this research. CONFLICT OF INTEREST STATEMENT Dr Michael J. Mirro has research funding from the Agency for Healthcare Research and Quality and the following financial relationships with industry to disclose: research grants from Biotronik, Inc., Medtronic, Inc., and Janssen Scientific Affairs, LLC; consulting fees/honoraria with McKesson Corporation, iRhythm Technologies, Inc., and Zoll Medical Corporation; and financial partnership with Medical Informatics Engineering, outside the submitted work. Dr Michael J. Mirro’s relationships with academia include serving as trustee of Indiana University and on the Indiana University Health Board. Dr Tammy Toscos has research funding from the Agency for Healthcare Research and Quality and the following financial relationships with industry to disclose: research grants from Biotronik, Inc., Medtronic, Inc., and Janssen Scientific Affairs, LLC; and iRhythm Technologies Inc. outside the submitted work. REFERENCES 1 Banova B. The impact of technology on healthcare. American Institute of Medical Sciences and Education (AIMS); 2018 . https://www.aimseducation.edu/blog/the-impact-of-technology-on-healthcare/. Accessed January 6, 2019. 2 Anderson M. Digital Divide Persists Even as Lower-Income Americans Make Gains in Tech Adoption . Washington, DC : Pew Research Center ; 2017 . http://www.pewresearch.org/fact-tank/2017/03/22/digital-divide-persists-even-as-lower-income-americans-make-gains-in-tech-adoption/. Accessed January 6, 2019. Google Preview WorldCat COPAC 3 Ginossar T , Nelson S. Reducing the health and digital divides: a model for using community-based participatory research approach to e-health interventions in low-income Hispanic communities . J Comput-Mediated Commun 2010 ; 15 ( 4 ): 530 – 551 . Google Scholar Crossref Search ADS WorldCat 4 Veinot TC , Mitchell H , Ancker JS. Good intentions are not enough: how informatics interventions can worsen inequality . JAMIA 2018 ; 25 ( 8 ): 1080 – 1088 . Google Scholar PubMed WorldCat 5 Jacobs M , Cramer H , Barkhuus L. Caring about sharing: couples’ practices in single user device access. In: Proceedings of the 19th Internatinal Conference on Supporting Group Work . New York, NY : ACM ; 2016 : 235 – 243 . Google Preview WorldCat COPAC 6 Matthews T , Liao K , Turner A , Berkovich M , Reeder R , Consolvo S. “She’ll just grab any device that’s closer”: a study of everyday device & account sharing in households In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems . New York, NY : ACM ; 2016 : 5921 – 5932 . Google Preview WorldCat COPAC 7 López CM , Qanungo S , Jenkins CM , Acierno R. Technology as a means to address disparities in mental health research: a guide to “tele-tailoring” your research methods . Prof Psychol Res Pract 2018 ; 49 ( 1 ): 57 – 64 . Google Scholar Crossref Search ADS WorldCat 8 Mirro M , Daley C , Wagner S , Rohani Ghahari R , Drouin M , Toscos T. Delivering remote monitoring data to patients with implantable cardioverter‐defibrillators: does medium matter? Pacing Clin Electrophysiol 2018 ; 41 ( 11 ): 1526 – 1535 . Google Scholar Crossref Search ADS PubMed WorldCat 9 Hibbard JH , Mahoney ER , Stockard J , Tusler M. Development and testing of a short form of the patient activation measure . Health Serv Res 2005 ; 40 ( 6 Pt 1 ): 1918 – 1930 . Google Scholar Crossref Search ADS PubMed WorldCat 10 Anderson-Lewis C , Darville G , Mercado RE , Howell S , Di Maggio S. mHealth technology use and implications in historically underserved and minority populations in the United States: systematic literature review . JMIR Mhealth Uhealth 2018 ; 6 ( 6 ): e128 . Google Scholar Crossref Search ADS PubMed WorldCat © The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Journal

Journal of the American Medical Informatics AssociationOxford University Press

Published: Aug 1, 2019

There are no references for this article.