Journal of the American Medical Informatics Association

Brunyé, Tad T; Booth, Kelsey; Hendel, Dalit; Kerr, Kathleen F; Shucard, Hannah; Weaver, Donald L; Elmore, Joann G

doi: 10.1093/jamia/ocad232pmid: 38031453

ObjectiveThis study explores the feasibility of using machine learning to predict accurate versus inaccurate diagnoses made by pathologists based on their spatiotemporal viewing behavior when evaluating digital breast biopsy images.Materials and MethodsThe study gathered data from 140 pathologists of varying experience levels who each reviewed a set of 14 digital whole slide images of breast biopsy tissue. Pathologists’ viewing behavior, including zooming and panning actions, was recorded during image evaluation. A total of 30 features were extracted from the viewing behavior data, and 4 machine learning algorithms were used to build classifiers for predicting diagnostic accuracy.ResultsThe Random Forest classifier demonstrated the best overall performance, achieving a test accuracy of 0.81 and area under the receiver-operator characteristic curve of 0.86. Features related to attention distribution and focus on critical regions of interest were found to be important predictors of diagnostic accuracy. Further including case-level and pathologist-level information incrementally improved classifier performance.DiscussionResults suggest that pathologists’ viewing behavior during digital image evaluation can be leveraged to predict diagnostic accuracy, affording automated feedback and decision support systems based on viewing behavior to aid in training and, ultimately, clinical practice. They also carry implications for basic research examining the interplay between perception, thought, and action in diagnostic decision-making.ConclusionThe classifiers developed herein have potential applications in training and clinical settings to provide timely feedback and support to pathologists during diagnostic decision-making. Further research could explore the generalizability of these findings to other medical domains and varied levels of expertise.

Kasun, Max; Ryan, Katie; Paik, Jodi; Lane-McKinley, Kyle; Dunn, Laura Bodin; Roberts, Laura Weiss; Kim, Jane Paik

doi: 10.1093/jamia/ocad238pmid: 38069455

ObjectivesWe set out to describe academic machine learning (ML) researchers’ ethical considerations regarding the development of ML tools intended for use in clinical care.Materials and MethodsWe conducted in-depth, semistructured interviews with a sample of ML researchers in medicine (N = 10) as part of a larger study investigating stakeholders’ ethical considerations in the translation of ML tools in medicine. We used a qualitative descriptive design, applying conventional qualitative content analysis in order to allow participant perspectives to emerge directly from the data.ResultsEvery participant viewed their algorithm development work as holding ethical significance. While participants shared positive attitudes toward continued ML innovation, they described concerns related to data sampling and labeling (eg, limitations to mitigating bias; ensuring the validity and integrity of data), and algorithm training and testing (eg, selecting quantitative targets; assessing reproducibility). Participants perceived a need to increase interdisciplinary training across stakeholders and to envision more coordinated and embedded approaches to addressing ethics issues.Discussion and ConclusionParticipants described key areas where increased support for ethics may be needed; technical challenges affecting clinical acceptability; and standards related to scientific integrity, beneficence, and justice that may be higher in medicine compared to other industries engaged in ML innovation. Our results help shed light on the perspectives of ML researchers in medicine regarding the range of ethical issues they encounter or anticipate in their work, including areas where more attention may be needed to support the successful development and integration of medical ML tools.

Smith, Joshua C; Williamson, Brian D; Cronkite, David J; Park, Daniel; Whitaker, Jill M; McLemore, Michael F; Osmanski, Joshua T; Winter, Robert; Ramaprasan, Arvind; Kelley, Ann; Shea, Mary; Wittayanukorn, Saranrat; Stojanovic, Danijela; Zhao, Yueqin; Toh, Sengwee;

Wu, JunBo; Yuan, Christina T; Moyal-Smith, Rachel; Wick, Elizabeth C; Rosen, Michael A

doi: 10.1093/jamia/ocad237pmid: 38078843

ObjectivesEnhanced recovery pathways (ERPs) are evidence-based approaches to improving perioperative surgical care. However, the role of electronic health records (EHRs) in their implementation is unclear. We examine how EHRs facilitate or hinder ERP implementation.Materials and MethodsWe conducted interviews with informaticians and clinicians from US hospitals participating in an ERP implementation collaborative. We used inductive thematic analysis to analyze transcripts and categorized hospitals into 3 groups based on process measure adherence. High performers exhibited a minimum 80% adherence to 6 of 9 metrics, high improvers demonstrated significantly better adherence over 12 months, and strivers included all others. We mapped interrelationships between themes using causal loop diagrams.ResultsWe interviewed 168 participants from 8 hospitals and found 3 thematic clusters: (1) “EHR difficulties” with the technology itself and contextual factors related to (2) “EHR enablers,” and (3) “EHR barriers” in ERP implementation. Although all hospitals experienced issues, high performers and improvers successfully integrated ERPs into EHRs through a dedicated multidisciplinary team with informatics expertise. Strivers, while enacting some fixes, were unable to overcome individual resistance to EHR-supported ERPs.Discussion and ConclusionWe add to the literature describing the limitations of EHRs’ technological capabilities to facilitate clinical workflows. We illustrate how organizational strategies around engaging motivated clinical teams with informatics training and resources, especially with dedicated technical support, moderate the extent of EHRs’ support to ERP implementation, causing downstream effects for hospitals to transform technological challenges into care-improving opportunities. Early and consistent involvement of informatics expertise with frontline EHR clinician users benefited the efficiency and effectiveness of ERP implementation and sustainability.

Lawrence, Joanna; South, Mike; Hiscock, Harriet; Capurro, Daniel; Sharma, Anurag; Ride, Jemimah

doi: 10.1093/jamia/ocad239pmid: 38078841

ObjectivesHospital costs continue to rise unsustainably. Up to 20% of care is wasteful including low value care (LVC). This study aimed to understand whether electronic medical record (EMR) alerts are effective at reducing pediatric LVC and measure the impact on hospital costs.Materials and MethodsUsing EMR data over a 76-month period, we evaluated changes in 4 LVC practices following the implementation of EMR alerts, using time series analysis to control for underlying time-based trends, in a large pediatric hospital in Australia. The main outcome measure was the change in rate of each LVC practice. Balancing measures included the rate of alert adherence as a proxy measure for risk of alert fatigue. Hospital costs were calculated by the volume of LVC avoided multiplied by the unit costs. Costs of the intervention were calculated from clinician and analyst time required.ResultsAll 4 LVC practices showed a statistically significant reduction following alert implementation. Two LVC practices (blood tests) showed an abrupt change, associated with high rates of alert adherence. The other 2 LVC practices (bronchodilator use in bronchiolitis and electrocardiogram ordering for sleeping bradycardia) showed an accelerated rate of improvement compared to baseline trends with lower rates of alert adherence. Hospital savings were $325 to $180 000 per alert.Discussion and ConclusionEMR alerts are effective in reducing pediatric LVC practices and offer a cost-saving opportunity to the hospital. Further efforts to leverage EMR alerts in pediatric settings to reduce LVC are likely to support future sustainable healthcare delivery.

Barwise, Amelia K; Curtis, Susan; Diedrich, Daniel A; Pickering, Brian W

doi: 10.1093/jamia/ocad224pmid: 38099504

ObjectivesInpatients with language barriers and complex medical needs suffer disparities in quality of care, safety, and health outcomes. Although in-person interpreters are particularly beneficial for these patients, they are underused. We plan to use machine learning predictive analytics to reliably identify patients with language barriers and complex medical needs to prioritize them for in-person interpreters.Materials and methodsThis qualitative study used stakeholder engagement through semi-structured interviews to understand the perceived risks and benefits of artificial intelligence (AI) in this domain. Stakeholders included clinicians, interpreters, and personnel involved in caring for these patients or for organizing interpreters. Data were coded and analyzed using NVIVO software.ResultsWe completed 49 interviews. Key perceived risks included concerns about transparency, accuracy, redundancy, privacy, perceived stigmatization among patients, alert fatigue, and supply–demand issues. Key perceived benefits included increased awareness of in-person interpreters, improved standard of care and prioritization for interpreter utilization; a streamlined process for accessing interpreters, empowered clinicians, and potential to overcome clinician bias.DiscussionThis is the first study that elicits stakeholder perspectives on the use of AI with the goal of improved clinical care for patients with language barriers. Perceived benefits and risks related to the use of AI in this domain, overlapped with known hazards and values of AI but some benefits were unique for addressing challenges with providing interpreter services to patients with language barriers.ConclusionArtificial intelligence to identify and prioritize patients for interpreter services has the potential to improve standard of care and address healthcare disparities among patients with language barriers.

Gao, Jianhui; Bonzel, Clara-Lea; Hong, Chuan; Varghese, Paul; Zakir, Karim; Gronsbell, Jessica

doi: 10.1093/jamia/ocad226pmid: 38128118

ObjectiveHigh-throughput phenotyping will accelerate the use of electronic health records (EHRs) for translational research. A critical roadblock is the extensive medical supervision required for phenotyping algorithm (PA) estimation and evaluation. To address this challenge, numerous weakly-supervised learning methods have been proposed. However, there is a paucity of methods for reliably evaluating the predictive performance of PAs when a very small proportion of the data is labeled. To fill this gap, we introduce a semi-supervised approach (ssROC) for estimation of the receiver operating characteristic (ROC) parameters of PAs (eg, sensitivity, specificity).Materials and MethodsssROC uses a small labeled dataset to nonparametrically impute missing labels. The imputations are then used for ROC parameter estimation to yield more precise estimates of PA performance relative to classical supervised ROC analysis (supROC) using only labeled data. We evaluated ssROC with synthetic, semi-synthetic, and EHR data from Mass General Brigham (MGB).ResultsssROC produced ROC parameter estimates with minimal bias and significantly lower variance than supROC in the simulated and semi-synthetic data. For the 5 PAs from MGB, the estimates from ssROC are 30% to 60% less variable than supROC on average.DiscussionssROC enables precise evaluation of PA performance without demanding large volumes of labeled data. ssROC is also easily implementable in open-source R software.ConclusionWhen used in conjunction with weakly-supervised PAs, ssROC facilitates the reliable and streamlined phenotyping necessary for EHR-based research.

Mullie, Louis; Afilalo, Jonathan; Archambault, Patrick; Bouchakri, Rima; Brown, Kip; Buckeridge, David L; Cavayas, Yiorgos Alexandros; Turgeon, Alexis F; Martineau, Denis; Lamontagne, François; Lebrasseur, Martine; Lemieux, Renald; Li, Jeffrey; Sauthier, Michaël; St-Onge, Pascal; Tang, An; Witteman, William; Chassé, Michaël

May, Sarah B; Giordano, Thomas P; Gottlieb, Assaf

doi: 10.1093/jamia/ocad217pmid: 37990631

ObjectiveThe HIV epidemic remains a significant public health issue in the United States. HIV risk prediction models could be beneficial for reducing HIV transmission by helping clinicians identify patients at high risk for infection and refer them for testing. This would facilitate initiation on treatment for those unaware of their status and pre-exposure prophylaxis for those uninfected but at high risk. Existing HIV risk prediction algorithms rely on manual construction of features and are limited in their application across diverse electronic health record systems. Furthermore, the accuracy of these models in predicting HIV in females has thus far been limited.Materials and methodsWe devised a pipeline for automatic construction of prediction models based on automatic feature engineering to predict HIV risk and tested our pipeline on a local electronic health records system and a national claims data. We also compared the performance of general models to female-specific models.ResultsOur models obtain similarly good performance on both health record datasets despite difference in represented populations and data availability (AUC = 0.87). Furthermore, our general models obtain good performance on females but are also improved by constructing female-specific models (AUC between 0.81 and 0.86 across datasets).Discussion and conclusionsWe demonstrated that flexible construction of prediction models performs well on HIV risk prediction across diverse health records systems and perform as well in predicting HIV risk in females, making deployment of such models into existing health care systems tangible.

Antonio, Marcy G; Veinot, Tiffany C

doi: 10.1093/jamia/ocad234pmid: 38134954

ObjectivesInvestigate how people with chronic obstructive pulmonary disease (COPD)—an example of a progressive, potentially fatal illness—are using digital technologies (DTs) to address illness experiences, outcomes and social connectedness.Materials and MethodsA transformative mixed methods study was conducted in Canada with people with COPD (n = 77) or with a progressive lung condition (n = 6). Stage-1 interviews (n = 7) informed the stage-2 survey. Survey responses (n = 80) facilitated the identification of participants for stage-3 interviews (n = 13). The interviews were thematically analyzed. Descriptive statistics were calculated for the survey. The integrative mixed method analysis involved mixing between and across the stages.ResultsMost COPD participants (87.0%) used DTs. However, few participants frequently used DTs to self-manage COPD. People used DTs to seek online information about COPD symptoms and treatments, but lacked tailored information about illness progression. Few expressed interest in using DTs for self- monitoring and tracking. The regular use of DTs for intergenerational connections may facilitate leaving a legacy and passing on traditions and memories. Use of DTs for leisure activities provided opportunities for connecting socially and for respite, reminiscing, distraction and spontaneity.Discussion and ConclusionWe advocate reconceptualizing consumer health technologies to prioritize quality of life for people with a progressive, potentially fatal illness. “Quality of life informatics” should focus on reducing stigma regarding illness and disability and taboo towards death, improving access to palliative care resources and encouraging experiences to support social, emotional and mental health. For DTs to support people with fatal, progressive illnesses, we must expand informatics strategies to quality of life.