Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests

Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Key Points Question How prevalent are low-yield IMPORTANCE Laboratory testing is an important target for high-value care initiatives, constituting inpatient diagnostic laboratory tests for the highest volume of medical procedures. Prior studies have found that up to half of all inpatient which results are predictable with laboratory tests may be medically unnecessary, but a systematic method to identify these machine learning models? unnecessary tests in individual cases is lacking. Findings In this diagnostic study of 191 506 inpatients from 3 tertiary OBJECTIVE To systematically identify low-yield inpatient laboratory testing through personalized academic medical centers, common predictions. low-yield inpatient diagnostic laboratory test results were systematically DESIGN, SETTING, AND PARTICIPANTS In this retrospective diagnostic study with multivariable identified through data-driven methods prediction models, 116 637 inpatients treated at Stanford University Hospital from January 1, 2008, and personalized predictions. to December 31, 2017, a total of 60 929 inpatients treated at University of Michigan from January 1, 2015, to December 31, 2018, and 13 940 inpatients treated at the University of California, San Meaning The findings suggest that Francisco from January 1 to December 31, 2018, were assessed. data-driven methods can make explicit the level of uncertainty and expected MAIN OUTCOMES AND MEASURES Diagnostic accuracy measures, including sensitivity, specificity, information gain from diagnostic tests, negative predictive values (NPVs), positive predictive values (PPVs), and area under the receiver with the potential to encourage useful operating characteristic curve (AUROC), of machine learning models when predicting whether testing and discourage low-value testing inpatient laboratory tests yield a normal result as defined by local laboratory reference ranges. that can incur direct cost and indirect harm. RESULTS In the recent data sets (July 1, 2014, to June 30, 2017) from Stanford University Hospital (including 22 664 female inpatients with a mean [SD] age of 58.8 [19.0] years and 22 016 male Supplemental content inpatients with a mean [SD] age of 59.0 [18.1] years), among the top 20 highest-volume tests, 792 397 were repeats of orders within 24 hours, including tests that are physiologically unlikely to Author affiliations and article information are listed at the end of this article. yield new information that quickly (eg, white blood cell differential, glycated hemoglobin, and serum albumin level). The best-performing machine learning models predicted normal results with an AUROC of 0.90 or greater for 12 stand-alone laboratory tests (eg, sodium AUROC, 0.92 [95% CI, 0.91-0.93]; sensitivity, 98%; specificity, 35%; PPV, 66%; NPV, 93%; lactate dehydrogenase AUROC, 0.93 [95% CI, 0.93-0.94]; sensitivity, 96%; specificity, 65%; PPV, 71%; NPV, 95%; and troponin I AUROC, 0.92 [95% CI, 0.91-0.93]; sensitivity, 88%; specificity, 79%; PPV, 67%; NPV, 93%) and 10 common laboratory test components (eg, hemoglobin AUROC, 0.94 [95% CI, 0.92-0.95]; sensitivity, 99%; specificity, 17%; PPV, 90%; NPV, 81%; creatinine AUROC, 0.96 [95% CI, 0.96-0.97]; sensitivity, 93%; specificity, 83%; PPV, 79%; NPV, 94%; and urea nitrogen AUROC, 0.95 [95% CI, 0.94, 0.96]; sensitivity, 87%; specificity, 89%; PPV, 77%; NPV 94%). CONCLUSIONS AND RELEVANCE The findings suggest that low-yield diagnostic testing is common and can be systematically identified through data-driven methods and patient context–aware predictions. Implementing machine learning models appear to be able to quantify the level of uncertainty and expected information gained from diagnostic tests explicitly, with the potential to (continued) Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 1/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Abstract (continued) encourage useful testing and discourage low-value testing that incurs direct costs and indirect harms. JAMA Network Open. 2019;2(9):e1910967. Corrected on October 11, 2019. doi:10.1001/jamanetworkopen.2019.10967 Introduction 1,2 Unsustainable growth in health care costs is exacerbated by waste that does not improve health. The Institute of Medicine estimates that more than $200 billion a year is spent on unnecessary tests and procedures. Given this amount of misallocated resources, there has been an increasing emphasis on high-value care, notably with the American Board of Internal Medicine Foundation’s Choosing Wisely guidelines. Laboratory testing, in particular, constitutes the highest-volume medical procedure, with estimates of up to 25% to 50% of all inpatient testing being medically 6,7 unnecessary. The consequences of unnecessary testing are not simply financial but also include low patient satisfaction, sleep fragmentation, risk of delirium, iatrogenic anemia, and increased 8-11 mortality. Numerous interventions have been studied to reduce inappropriate laboratory testing, including clinical education, audit feedback, financial incentives, and electronic medical record 12-15 (EMR)–based ordering restrictions. Interventions based on EMRs offer pertinent information for clinical decision-making, such as cost, turnaround time, prior stable results, and guideline-based best 16-20 practice alerts. Despite these efforts, unnecessary tests remain prolific when practitioners are influenced by fear of missing problems, medicolegal concerns, patient preferences, and the overall difficulty of systematically identifying low-value testing at the point of care, prompting behavior to 21,22 check just in case. We envisioned patient-specific estimates of the pretest probability of results for any diagnostic test, displayed at the point of clinical order entry. When humans tend to have poor intuition for estimating probabilities and diagnostic test performance, having automated computer systems explicitly provide those estimates could substantially change clinical practice. Machine learning in medicine now offers a direct mechanism to produce such estimates by predicting select laboratory 24-30 results. Although prior approaches can provide a laboratory result given other simultaneously available results (eg, estimating ferritin levels when other components of an iron panel are given), this is too late for decision support to change behavior when the tests are already performed. We addressed the more clinically relevant question of predicting laboratory results with only information available before the test is ordered. Our objective was to identify inpatient diagnostic laboratory testing with predictable results that are unlikely to yield new information. Our analytic approach escalated from descriptive statistics to machine learning models for individualized estimates of predictable test results. Methods This diagnostic study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline for reporting results of multivariate prediction models to develop and evaluate our machine learning methods (eFigure 1 in the Supplement gives an overview of our approach). Ten years (January 1, 2008, to December 31, 2017) of inpatient electronic medical record (EMR) data from hospitals at Stanford University, 4 years (January 1, 2015, to December 31, 2018) of data from University of Michigan (UMich), and 1 year (2018) of data from University of California, San Francisco (UCSF) were used for this study. To preserve data privacy, raw clinical data were deidentified, processed, trained, and evaluated locally at each local site, with only evaluation results sent back to Stanford for further analysis. The Stanford University, UMich, and UCSF JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 2/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests institutional review boards approved the study at each site. Project-specific informed consent was not required because the study was restricted to secondary analysis of existing clinical data. Patient data at Stanford University were extracted and deidentified by the STRIDE (Stanford Translational Research Integrated Database Environment) project, a research and development project at Stanford University to create a standards-based informatics platform supporting clinical and translational research. Participants and Inclusion Criteria All laboratory test results had reference labels for normal vs abnormal results as defined by local clinical laboratory reference ranges and at least 500 occurrences in a data set. For each laboratory test, we retrieved a random sample of 10 000 test orders from the available data (or all orders if <10 000). Outcome and Evaluation Metrics Our goal was to predict the result (negative vs positive) of each laboratory test using information available before the order was placed. We considered stand-alone tests, in which a single order yielded a single result (eg, magnesium level, lactate level, or blood cultures), and panel tests that yielded multiple component results (eg, a complete blood cell count panel yielded white blood cell, hemoglobin, and platelet component results). We predicted the results of each panel component separately to avoid labeling an entire panel as positive or negative. We evaluated prediction performance through standard metrics for diagnostic accuracy, including the area under the receiver operating characteristic curve (ie, AUROC or C statistic), which summarizes the trade-off between sensitivity and specificity. Given specific decision thresholds, we calculated diagnostic test metrics, including sensitivity, specificity, positive predictive value, and negative predictive value (NPV). Typically, such metrics evaluate how well a test predicts a diagnosis. In our case, a test result being abnormal was itself the diagnosis, whereas the prediction algorithms operated as screening tests compared with the physical laboratory tests. For example, NPV was the probability of being correct when a negative or normal result was predicted. Predictors and Data Feature For each laboratory, 875 raw features from the Stanford University EMR that reflected patient clinical context available at the time of the order entry were extracted (eTable 1 in the Supplement). The core features included patient demographics, normality of the most recent test of interest, numbers of recent tests of interest, history of Charlson Comorbidity Index categories, which specialty team was treating the patient, time since admission, time of day and year of the test, and summary statistics of recent vital statistics and laboratory results. Vital statistics and treatment team information were not accessible in the UMich data sample, which yielded 603 raw features. Age and sex information were not accessible in the UCSF data sample, which yielded 806 raw features. Development vs Validation Split Patients were randomly split into training (development) and held-out test (validation) sets with a 75:25 split. The model was developed based on the training data alone but assessed generalizable predictive accuracy on the separate patients in the held-out sets. Missing Data Most of the data features, such as history of a comorbidity category or the number of prior laboratory tests, always had a valid value (including not present or zero). Numerical results (eg, mean sodium level in the past week) could be missing, in which case we carried forward the most recent value from the patient’s prior records. If no prior values existed, we imputed the training sample mean. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 3/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Feature Selection We applied recursive feature elimination (with cross-validation) to select the top 5% most important features for model building that best improved accuracy when included in prediction models. This resulted in 43 processed features in each subsequent prediction model (the eMethods in the Supplement gives technical explanations). Model Development We built an array of prediction models using established algorithms, including regularized logistic regression, regress and round, naive Bayes, neural network multilayer perceptrons, decision tree, random forest, AdaBoost, and XGBoost. Each model generated a prediction score between 0 and 1 for how likely a laboratory test result would be negative or normal vs positive or abnormal (Figure 1A). A baseline model predicted the most recent result (if the patient had a prior test) or the Figure 1. Normality Scores, Decision Threshold, and Receiver Operating Characteristic (ROC) Curve A Before thresholding B After thresholding 800 800 Abnormal False negative Normal True positive True negative 600 600 False positive 400 400 200 200 0 0 0 0.2 0.4 0.6 0.8 1.0 0 0.2 0.4 0.6 0.8 1.0 Score Score C Threshold on ROC 1.0 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1.0 1–Specificity A, Histogram of normality scores distributed among normal and abnormal orders. B, false-positive orders were predicted to be abnormal but actually were normal, and true- After picking a threshold, orders were classified as predicted to be normal if their positive orders were predicted to be abnormal but actually were abnormal. C, This choice normality scores were above the threshold or predicted to be abnormal if they were of threshold led to a sensitivity of 96% and specificity of 67%, as shown on the below the threshold. True-negative orders were predicted to be normal but actually were ROC curve. normal, false-negative orders were predicted to be normal but actually were abnormal, JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 4/13 Sensitivity No. of Orders No. of Orders JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests overall prevalence of positive results as the prediction score. Additional model specifications are included in eTable 2 in the Supplement. Decision Threshold Estimation Decision thresholds translate continuous prediction scores into discrete negative vs positive predictions (Figure 1B). We conservatively favored high sensitivity and high NPV to minimize the risks 35,36 of alert fatigue and missing clinically important laboratory test result abnormalities (Figure 1C). This article gives results targeting an NPV of 95%, recognizing the diminishing returns of expected information gain when one is already 95% certain of the result. These diminishing returns were easily adjusted for different clinical scenarios with varying tolerances for uncertainty because we confirmed robustness across a range of options targeting NPVs of 99%, 95%, 90%, and 80% (eTable 3 in the Supplement). Statistical Analysis To assess the statistical significance of the results, we calculated 95% CIs for AUROCs by resampling the evaluation set 1000 times for each laboratory (Table and eTables 3-8 in the Supplement). We performed additional randomized permutation tests to compare the AUROC of the best-performing algorithm against that of the baseline model (eFigures 2-7 in the Supplement). Multisite Evaluation We performed equivalent analysis from multiple sites, including hospitals at Stanford University, UMich, and UCSF. We developed mapping software between the data formats from different sites to allow for a common analytic process at each site without sharing raw clinical data. We cross- evaluated performances of models trained at one site and then tested at another. Results Prevalence of Repetitive Tests The recent data sets (July 1, 2014, to June 30, 2017) from Stanford University Hospital included 22 664 female inpatients (mean [SD] age, 58.8 [19.0] years) and 22 016 male inpatients (mean [SD] Table. Diagnostic Performance Metrics of Top-Volume Stand-Alone Laboratory Tests Predicting Whether Laboratory Tests Will Yield a Normal Result on a Held-Out Evaluation Set, Targeting at an NPV of 95% No. of Orders/ Metric, % 1000 Patient Laboratory Test Encounters AUROC (95% CI) Prevalence NPV PPV Sensitivity Specificity TN FN TP FP Magnesium 4246 0.76 (0.74-0.78) 26 91 36 86 47 35 3.6 22 39 Prothrombin time 2244 0.89 (0.88-0.91) 80 85 81 100 3.6 0.7 0.1 80 19 Phosphorus 2120 0.74 (0.72-0.76) 33 88 39 91 30 20 2.8 30 47 Partial thromboplastin 1471 0.86 (0.85-0.87) 61 87 65 98 17 6.5 1.0 60 32 time Lactate 1230 0.87 (0.85-0.88) 29 91 56 82 74 53 5.2 23 19 Calcium, ionized 1197 0.72 (0.70-0.74) 61 90 62 100 4.8 1.9 0.2 61 37 Potassium 752 0.81 (0.79-0.84) 12 92 40 43 91 80 7.0 5.2 7.9 Troponin I 534 0.92 (0.91-0.93) 33 93 67 88 79 53 4.0 29 14 LDH 455 0.93 (0.93-0.94) 47 95 71 96 65 35 1.8 45 18 Blood culture Aerobic and anaerobic 400 0.66 (0.61-0.71) 8.1 93 16 16 93 85 6.8 1.3 6.6 2 Aerobic 371 0.62 (0.58-0.67) 9.1 93 12 61 54 49 3.6 5.6 42 Sodium 361 0.92 (0.91-0.93) 57 93 66 98 35 15 1.1 56 28 Abbreviations: AUROC, area under the receiver operating characteristic curve; FN, false- Prevalence of abnormal or positive test results. negative; FP, false-positive; LDH, lactate dehydrogenase; NPV, negative predictive value; PPV, positive predictive value; TN, true-negative; TP, true-positive. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 5/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests age, 59.0 [18.1] years). Figure 2A reports the overall volume of the most commonly repeated inpatient laboratory tests at Stanford University during that period. Among the top 20 volume tests, 792 397 were repeats of orders within 24 hours. Figure 2B reports the repetition rate of common tests medically implausible to yield new information from frequent testing (eg, glycated hemoglobin). The likelihood of common laboratory components that yielded a negative result progressively increased as repeated negative results were observed (Figure 2C). Model Performance Random forest and XGBoost demonstrated the highest discriminating power for most of the stand- alone laboratory tests from Stanford University hospital, yielding a mean AUROC of 0.77 compared with 0.67 with the baseline model (eFigure 2 in the Supplement gives the ROC curves). The best- performing machine learning models predicted normal results with an AUROC of 0.90 or greater for 12 stand-alone laboratory tests (eg, sodium AUROC, 0.92 [95% CI, 0.91-0.93]; sensitivity, 98%; specificity, 35%; PPV, 66%; NPV, 93%; lactate dehydrogenase AUROC, 0.93 [95% CI, 0.93-0.94]; sensitivity, 96%; specificity, 65%; PPV, 71%; NPV, 95%; and troponin I AUROC, 0.92 [95% CI, 0.91- 0.93]; sensitivity, 88%; specificity, 79%; PPV, 67%; NPV, 93%) and 10 common laboratory test components (eg, hemoglobin AUROC, 0.94 [95% CI, 0.92-0.95]; sensitivity, 99%; specificity, 17%; PPV, 90%; NPV, 81%; creatinine AUROC, 0.96 [95% CI, 0.96-0.97]; sensitivity, 93%; specificity, 83%; PPV, 79%; NPV, 94%; and urea nitrogen AUROC, 0.95 [95% CI, 0.94, 0.96]; sensitivity, 87%; specificity, 89%; PPV, 77%; NPV 94%). Diagnostic performance metrics for the most common stand- alone laboratory tests when targeting 95% NPV are given in the Table, with the full table of all Figure 2. Prevalence of Repetitive Tests and Their Diminishing Information Gain <24 h 24 h to 3 d 3 d to 1 wk >1 wk A B Highly repeated common tests Guideline-controlled tests C-peptide Basic metabolic panel Albumin Magnesium Lipase CBC with diff Aspergillus galactomannan Prothrombin time C-reactive protein Phosphorus Glycated hemoglobin PTT NT-proBNP Lactate Sedimentation rate Calcium ionized Thyrotropin Potassium 0 20 40 60 80 100 Troponin I LDH total Tests, % Heparin activity level Sodium C Consecutive normal test findings in past 7 days Lidocaine level Hematocrit 1.0 Uric acid 0.8 Sepsis protocol lactate Lactic acid White blood cells 0.6 Fibrinogen Hemoglobin Specific gravity 0.4 Sodium Potassium 0 1000 2000 3000 4000 5000 0.2 Creatinine No. of Orders per 1000 Patient Encounters 0 1 2 3 4 5 Consecutive Normal Test Findings in Past 7 Days A, Most commonly repeated laboratory test orders from July 1, 2014, to June 30, 2017, and 6.7% of glycated hemoglobin inpatient tests were performed again within 24 hours, at Stanford University Hospital. The total length of each bar represents the total volume even when it was not biologically plausible for the results to meaningfully change that of laboratory orders per 1000 patient encounters, with shaded regions reflecting how rapidly. C, Prevalence of normal results for common laboratory components many of these were repeated orders within a given time. For example, 47% of basic progressively increased toward 100% as more subsequent normal results were observed metabolic panels were subsequent tests performed again within 24 hours of the past in the prior week. CBC with diff indicates complete blood cell count with differential; order. Results were sorted by this number of repeated tests within 24 hours. B, LDH, lactate dehydrogenase; NT-proBNP, N-terminal pro–brain-type natriuretic peptide; Distribution of repeated orders for laboratory tests specifically identified as rarely ever and PTT, partial thromboplastin time. having clinical justification for repeated daily testing. For example, 18.3% of albumin JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 6/13 Normal Rate JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests laboratory tests evaluated in eTable 3 in the Supplement. Performance metrics for the common components in complete blood cell counts and comprehensive metabolic panels are given in Figure 3 along with results from UMich and UCSF data, with the full table of diagnostic performance metrics in eTable 4 in the Supplement and ROC curves in eFigure 3 in the Supplement. Model Transferability The respective prediction results for UMich data are reported in eFigure 4, eFigure 5, eTable 5, and eTable 6 in the Supplement, whereas similar results from UCSF are reported in eFigure 6, eFigure 7, eTable 7, and eTable 8 in the Supplement. Figure 4 gives the performance of models trained at Stanford University and subsequently evaluated at all sites. Although cross-site performance declined compared with local performance (eg, when predicting albumin results, AUROC decreased from 0.92 [95% CI, 0.91-0.94] when locally tested at Stanford University to 0.73 [95% CI, 0.70- 0.75] when remotely tested at UMich), predictive power was retained (AUROC, >0.85) for most Figure 3. Diagnostic Metrics of Predictions on Common Components in the Data Sets of Stanford University, University of Michigan (UMich), and University of California, San Francisco (UCSF) False positives True positives True negatives False negatives Fractions of true-negative, false-negative, false- Stanford UMich UCSF positive, and true-positive results are scaled by the Predicted Predicted Predicted Predicted Predicted Predicted number of orders among each 1000 patient Abnormal Normal Abnormal Normal Abnormal Normal encounters. Predicted normal represents the volume White blood cells that the model would suggest not to order, and we Hemoglobin targeted to limit the fraction of false-negative results Platelets Sodium to less than 5%. For some laboratory tests (eg, albumin Potassium measurement at Stanford University), there were Carbon dioxide almost zero predicted normal results, which means Urea nitrogen Creatinine that a few orders existed in the training set that were Calcium unpredictable; thus, the predictor could not Albumin confidently achieve a 95% negative predictive value Protein Alkaline phosphatase by picking any threshold above 0. The model chose a Total bilirubin decision threshold equal to 0, which led to scores of all AST orders in the test set falling above the decision ALT threshold, thus always encouraging ordering the test. 8000 6000 3000 0 3000 6000 6000 3000 0 3000 6000 6000 3000 0 3000 6000 ALT indicates alanine aminotransferase; AST, aspartate No. of Orders per No. of Orders per No. of Orders per 1000 Patient Encounters 1000 Patient Encounters 1000 Patient Encounters aminotransferase. Figure 4. Area Under the Receiver Operating Characteristic Curve (AUROC) Scores of Models for 15 Common Laboratory Test Components Developed at Stanford University but Evaluated at All Sites White blood cells UMich Hemoglobin UCSF Platelets Stanford Sodium Potassium Carbon dioxide Urea nitrogen Creatinine Calcium Albumin Protein The model generally achieved highest performance Alkaline phosphatase when evaluated locally at Stanford University with an Total bilirubin AUROC of 0.9 or greater for 10 laboratory test AST components but still retained at 0.85 or greater in 9 ALT cases when evaluated remotely at University of 0.5 0.6 0.7 0.8 0.9 1.0 California, San Francisco (UCSF) and University of AUROC Score Michigan (UMich). JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 7/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests laboratory components (eTable 9 in the Supplement gives the full comparison data). For certain tests, such as sodium level, however, the model trained at Stanford University had a better AUROC when tested at UMich (0.91; 95% CI, 0.90-0.93) than locally at Stanford University (0.87; 95% CI, 0.85-0.88). Inspection of the data and model showed that the UMich sodium level was easier to predict, with a baseline model already yielding an AUROC of 0.87 at UMich and 0.79 at Stanford University. Discussion Interpretation This study systematically identified low-yield diagnostic laboratory tests. Starting with simple descriptive statistics, Figure 2 shows how frequently laboratory tests are performed again. Although some tests may have credible reasons for such frequent repetition, guidelines and external knowledge can help identify some low-value repeated tests. For example, hundreds of tests for serum albumin, thyrotropin, and glycated hemoglobin levels were performed again within 24 hours, along with tens of thousands of repetitive tests for phosphorus and complete blood cell counts with differential. This finding quantitatively supports issues suggested in previous guidelines that hospitals can immediately use to target unnecessary repeated tests, such as through best practice 12,19,38-40 alerts showing recently available test results. Most instances of low-yield testing are not as straightforward to identify; thus, our study added machine learning methods for personalized test result predictions. Additional features, such as patient demographics, vital signs, and other common laboratory results, can be synthesized through machine learning models to produce more robust and accurate predictions. Although different applications and clinical contexts will have different tolerances for uncertainty, the study gave the primary results when choosing a conservative target NPV close to 95% (when the model predicted a test result was going to be normal, the goal was for it to be correct 95% of the time). This approach fits a scenario in which these targets are implemented as best practice alerts with a desire to maintain a small number of false-positive results (5%). The results at this level of pretest were estimated by which pursuing further testing would yield markedly diminishing returns. eTables 3 through 8 in the Supplement give similar results across a range of different NPV targets. Consistent with existing guideline-based forms of clinical decision support, pretest estimates of whether a laboratory test result will be normal would inform physician decision-making but not dictate or replace it. Ultimately, medical testing decisions are always based on varying levels of diagnostic certainty, even if practitioners are only implicitly aware that they are empirically estimating probabilistic risks based on patient characteristics. For example, blood cultures are not performed for every febrile patient because a credible risk of bacteremia is qualitatively recognized in only certain situations. Likewise, blood cultures are performed in sets of 4 bottles at a time, but we do not continue to check 5 or more bottles because we recognize further repeated tests are unlikely to yield information that was not already predictable based on the prior results. This approach provides a systematic and quantitative way to inform such decisions. The results should encourage practitioners and quality improvement committees to make explicit and quantitative their own embedded assumptions on acceptable decision thresholds. The general framework presented to quantify uncertainty can then feed into individual point-of-care decisions or more formal decision analyses. Implications This study provides a general approach to identifying predictable laboratory tests. Many of the laboratory tests that we evaluated have been evaluated for overuse, including magnesium 15,43-45 46 47 level, blood cultures, and complete blood cell counts. Patient-specific estimates of laboratory test result normality at the point-of-order entry may discourage low-yield tests with predictably negative results and encourage appropriate tests with high levels of uncertainty. For JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 8/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests example, when our method did not predict a blood culture result to be negative, this corresponds to greater than 16% positive predictive value (Table). This finding is more than enough risk of bacteremia to prompt diagnostic testing and even empirical treatment. This approach can also raise questions on how guideline- and protocol-based testing is implemented and could be optimized. The optimal threshold of acceptable uncertainty depends on the clinical scenario and the particular test. For example, although screening tests (eg, HIV testing or pregnancy screens in hospital settings) have predictable normal results, most of the time, they are unlikely to be influenced by decision support when the effect of missing an abnormal case is sufficiently severe and driven by overriding protocols. Similarly, regulatory requirements around sepsis protocols are a major driver of repeated lactate testing that may not be amenable to decision support on predictable results. The results of this study can still inform the development of such regulatory requirements on the appropriate number and interval of screening tests that may otherwise be excessive or too rigid for individual cases. In predictable cases, the risk of false-positive test results (and adverse downstream effects) may be substantial. These results can also provide foundational quantitative support for cost-effectiveness analysis. For example, if scaling the annual volume of predictable tests (predicted normal results) by their financial costs (eTable 10 in the Supplement), one could estimate annual savings by avoiding these tests. However, this saving should be carefully compared against potential harms and costs generated from missing the actually abnormal tests (false-negative results). In cases of panel test ordering, practitioners are often only interested in 1 or 2 components of panel tests at a time (eg, sodium level from a metabolic panel or hemoglobin level from a complete blood cell count). Most panel components may be predictably normal, but there could still be value in the overall order if there is sufficient uncertainty in at least 1 other clinically relevant component. Our separate predictions for each panel component in Figure 3 would allow practitioners to decide which components are relevant for their decision-making in future point-of-care information displays. The results also allow us to systematically identify relevant factors that are predictive of each test result. This identification can inform simple rule-based clinical decision support based on factors including obvious elements, such as prior results, and less obvious ones, such as sex for ferritin status and surgical vs medical team for cerebrospinal fluid studies. eTables 11 through 16 in the Supplement include a full list of the most important features for predicting the normality of each laboratory test result. Limitations Although we used conservative fixed-decision thresholds for clarity (targeting 95% NPV) in this proof-of-concept study, specific applications can undergo explicit decision analysis to assess the balance between risk and benefit. Even then, such future studies would require the foundation that we have established to assess the relative likelihood of different testing outcomes. Assuming that the training data reflect the same distribution as the evaluation, intended application data distribution is an important limitation in any prediction model. Although we believe it may ultimately be more valuable to disseminate our underlying approach to undergo continuous learning and adaptation to local environments, we assessed model performance across multiple sites. Figure 4 shows that models trained at Stanford University can often still retain useful predictive performance when evaluated at UCSF and UMich, although these models will predictably underperform locally trained models. For example, the decrease in performance when predicting albumin levels at UMich with the model trained at Stanford University is likely associated with different underlying population distributions, including substantially different prevalences of normal albumin test results (16% at Stanford vs 57% at UMich). On the other hand, the surprising increase of AUROC when applying the sodium model trained at Stanford University to UMich may indicate that sodium level was more excessively tested at UMich, making it easier to identify predictable repeated tests in their data. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 9/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Another factor that may lead to prediction failure is that the distribution of data could change over time. The point of refining decision support systems is to change ordering behavior, which is itself one of the most useful inputs into the predictive models. Consequently, we would recommend online learning algorithms that continuously adapt to practice changes rather than ever expecting to have a completed final model. Conclusions The findings suggest that low-yield diagnostic testing is common and can be systematically identified through data-driven methods and patient context–aware predictions. Implementing continuous learning prediction models may help quantify the level of uncertainty and expected information gain from diagnostic tests explicitly, with potential to encourage useful testing and discourage low-value testing that can incur direct costs and indirect harms. ARTICLE INFORMATION Accepted for Publication: July 22, 2019. Published: September 11, 2019. doi:10.1001/jamanetworkopen.2019.10967 Correction: This article was corrected on October 11, 2019, to fix errors in Figure 2B, Figure 3, and Limitations. Open Access: This is an open access article distributed under the terms of the CC-BY License.©2019XuSetal. JAMA Network Open. Corresponding Author: Jonathan H. Chen, MD, PhD, Center for Biomedical Informatics Research, Department of Medicine, Stanford University, 1265 Welch Rd, Medical School Office Bldg X213, Stanford, CA 94305 (jonc101@ stanford.edu). Author Affiliations: Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California (Xu, Balasubramanian, Chen); Division of Hospital Medicine, Department of Medicine, Stanford University, Stanford, California (Hom, Chen); Department of Pathology, University of Michigan School of Medicine, Ann Arbor (Schroeder); Department of Medicine, University of California, San Francisco (Najafi); Department of Computer Science, Stanford University, Stanford, California (Roy). Author Contributions: Drs Xu and Hom contributed equally to this work. Drs Xu and Chen had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: Roy, Chen. Acquisition, analysis, or interpretation of data: Xu, Hom, Balasubramanian, Schroeder, Najafi, Chen. Drafting of the manuscript: Xu, Hom, Najafi, Roy, Chen. Critical revision of the manuscript for important intellectual content: Xu, Hom, Balasubramanian, Schroeder, Chen. Statistical analysis: Xu, Balasubramanian, Chen. Obtained funding: Chen. Administrative, technical, or material support: Balasubramanian, Schroeder, Chen. Supervision: Chen. Conflict of Interest Disclosures: Dr Chen reported receiving grants from the National Institute of Environmental Health Sciences and the Gordon and Betty Moore Foundation during the conduct of the study and having co-ownership of Reaction Explorer LLC (chemistry education software company). No other disclosures were reported. Funding/Support: This study was supported by National Institutes of Health Big Data 2 Knowledge Award K01ES026837 through the National Institute of Environmental Health Sciences and in part by grant GBMF8040 from the Gordon and Betty Moore Foundation (Dr Chen). The STRIDE project was supported by grant UL1 RR025744 from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health. Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 10/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or Stanford Health Care. REFERENCES 1. Alvin MD, Jaffe AS, Ziegelstein RC, Trost JC. Eliminating creatine kinase-myocardial band testing in suspected acute coronary syndrome: a value-based quality improvement. JAMA Intern Med. 2017;177(10):1508-1512. doi:10. 1001/jamainternmed.2017.3597 2. Dieleman JL, Squires E, Bui AL, et al. Factors associated with increases in US health care spending, 1996-2013. JAMA. 2017;318(17):1668-1678. doi:10.1001/jama.2017.15927 3. Smith M, Saunders R, Stuckhardt L, McGinnis JM, eds; Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: National Academies Press; 2014. 4. Colla CH, Morden NE, Sequist TD, Schpero WL, Rosenthal MB. Choosing Wisely: prevalence and correlates of low-value health care services in the United States. J Gen Intern Med. 2015;30(2):221-228. doi:10.1007/s11606- 014-3070-z 5. Zhi M, Ding EL, Theisen-Toupal J, Whelan J, Arnaout R. The landscape of inappropriate laboratory testing: a 15- year meta-analysis. PLoS One. 2013;8(11):e78962. doi:10.1371/journal.pone.0078962 6. Yeh DD. A clinician’s perspective on laboratory utilization management. Clin Chim Acta. 2014;427(January): 145-150. doi:10.1016/j.cca.2013.09.023 7. Konger RL, Ndekwe P, Jones G, et al. Reduction in unnecessary clinical laboratory testing through utilization management at a US government Veterans Affairs hospital. Am J Clin Pathol. 2016;145(3):355-364. doi:10.1093/ ajcp/aqv092 8. Koch CG, Li L, Sun Z, et al. Hospital-acquired anemia: prevalence, outcomes, and healthcare implications. JHosp Med. 2013;8(9):506-512. doi:10.1002/jhm.2061 9. Koch CG, Li L, Sun Z, et al. From bad to worse: anemia on admission and hospital-acquired anemia. J Patient Saf. 2017;13(4):211-216. doi:10.1097/PTS.0000000000000142 10. Thavendiranathan P, Bagai A, Ebidia A, Detsky AS, Choudhry NK. Do blood tests cause anemia in hospitalized patients? the effect of diagnostic phlebotomy on hemoglobin and hematocrit levels. J Gen Intern Med. 2005;20 (6):520-524. doi:10.1111/j.1525-1497.2005.0094.x 11. Salisbury AC, Reid KJ, Alexander KP, et al. Diagnostic blood loss from phlebotomy and hospital-acquired anemia during acute myocardial infarction. Arch Intern Med. 2011;171(18):1646-1653. doi:10.1001/archinternmed.2011.361 12. Eaton KP, Levy K, Soong C, et al. Evidence-based guidelines to eliminate repetitive laboratory testing. JAMA Intern Med. 2017;177(12):1833-1839. doi:10.1001/jamainternmed.2017.5152 13. Stammen LA, Stalmeijer RE, Paternotte E, et al. Training physicians to provide high-value, cost-conscious care: a systematic review. JAMA. 2015;314(22):2384-2400. doi:10.1001/jama.2015.16353 14. Hom J, Kumar A, Evans KH, et al. A high value care curriculum for interns: a description of curricular design, implementation and housestaff feedback. Postgrad Med J. 2017;93(1106):725-729. doi:10.1136/postgradmedj- 2016-134617 15. Han SJ, Saigal R, Rolston JD, et al. Targeted reduction in neurosurgical laboratory utilization: resident-led effort at a single academic institution. J Neurosurg. 2014;120(1):173-177. doi:10.3171/2013.8.JNS13512 16. Sedrak MS, Myers JS, Small DS, et al. Effect of a price transparency intervention in the electronic health record on clinician ordering of inpatient laboratory tests: the PRICE randomized clinical trial. JAMA Intern Med. 2017;177 (7):939-945. doi:10.1001/jamainternmed.2017.1144 17. Silvestri MT, Bongiovanni TR, Glover JG, Gross CP. Impact of price display on provider ordering: a systematic review. JHospMed. 2016;11(1):65-76. doi:10.1002/jhm.2500 18. Feldman LS, Shihab HM, Thiemann D, et al. Impact of providing fee data on laboratory test ordering: a controlled clinical trial. JAMA Intern Med. 2013;173(10):903-908. doi:10.1001/jamainternmed.2013.232 19. Jun T, Kwang H, Mou E, et al. An electronic best practice alert based on Choosing Wisely guidelines reduces thrombophilia testing in the outpatient setting. J Gen Intern Med. 2019;34(1):29-30. doi:10.1007/s11606-018- 4663-8 20. Felcher AH, Gold R, Mosen DM, Stoneburner AB. Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing. J Am Med Inform Assoc. 2017;24(4):776-780. doi: 10.1093/jamia/ocw182 21. Sedrak MS, Patel MS, Ziemba JB, et al. Residents’ self-report on why they order perceived unnecessary inpatient laboratory tests. JHospMed. 2016;11(12):869-872. doi:10.1002/jhm.2645 JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 11/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests 22. Miyakis S, Karamanof G, Liontos M, Mountokalakis TD. Factors contributing to inappropriate ordering of tests in an academic medical department and the effect of an educational feedback strategy. Postgrad Med J. 2006; 82(974):823-829. doi:10.1136/pgmj.2006.049551 23. Manrai AK, Bhatia G, Strymish J, Kohane IS, Jain SH. Medicine’s uncomfortable relationship with math: calculating positive predictive value. JAMA Intern Med. 2014;174(6):991-993. doi:10.1001/jamainternmed. 2014.1059 24. Luo Y, Szolovits P, Dighe AS, Baron JM. Using machine learning to predict laboratory test results. Am J Clin Pathol. 2016;145(6):778-788. doi:10.1093/ajcp/aqw064 25. Lidbury BA, Richardson AM, Badrick T. Assessment of machine-learning techniques on large pathology data sets to address assay redundancy in routine liver function test profiles. Diagnosis (Berl). 2015;2(1):41-51. doi:10. 1515/dx-2014-0063 26. Aikens RC, Balasubramanian S, Chen JH. A machine learning approach to predicting the stability of inpatient lab test results. AMIA Jt Summits Transl Sci Proc. 2019;2019:515-523. 27. Cabitza F, Banfi G. Machine learning in laboratory medicine: waiting for the flood? Clin Chem Lab Med. 2018; 56(4):516-524. doi:10.1515/cclm-2017-0287 28. Chen JH, Asch SM. Machine learning and prediction in medicine: beyond the peak of inflated expectations. N Engl J Med. 2017;376(26):2507-2509. doi:10.1056/NEJMp1702071 29. Obermeyer Z, Emanuel EJ. Predicting the future: big data, machine learning, and clinical medicine. N Engl J Med. 2016;375(13):1216-1219. doi:10.1056/NEJMp1606181 30. Beam AL, Kohane IS. Big data and machine learning in health care. JAMA. 2018;319(13):1317-1318. doi:10.1001/ jama.2017.18391 31. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162(1):55-63. doi:10. 7326/M14-0697 32. Hripcsak G. Visualizing the operating range of a classification system. J Am Med Inform Assoc. 2012;19(4): 529-532. doi:10.1136/amiajnl-2011-000633 33. Hastie T, Tibshirani R, Friedman J. The elements of statistical learning. Elements. 2009;1:337-387. 34. Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY: Association for Computing Machinery; 2016:785-784. 35. Backman R, Bayliss S, Moore D, Litchfield I. Clinical reminder alert fatigue in healthcare: a systematic literature review protocol using qualitative evidence. Syst Rev. 2017;6(1):255. doi:10.1186/s13643-017-0627-z 36. Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R; with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17(1):36. doi:10.1186/s12911-017-0430-8 37. Johnson HA. Diminishing returns on the road to diagnostic certainty. JAMA. 1991;265(17):2229-2231. doi:10. 1001/jama.1991.03460170083038 38. Krasowski MD, Chudzik D, Dolezal A, et al. Promoting improved utilization of laboratory testing through changes in an electronic medical record: experience at an academic medical center. BMC Med Inform Decis Mak. 2015;15(February):11. doi:10.1186/s12911-015-0137-7 39. Chen JH, Fang DZ, Tim Goodnough L, Evans KH, Lee Porter M, Shieh L. Why providers transfuse blood products outside recommended guidelines in spite of integrated electronic best practice alerts. JHospMed. 2015; 10(1):1-7. doi:10.1002/jhm.2236 40. Pageler NM, Franzon D, Longhurst CA, et al. Embedding time-limited laboratory orders within computerized provider order entry reduces laboratory utilization. Pediatr Crit Care Med. 2013;14(4):413-419. doi:10.1097/PCC. 0b013e318272010c 41. Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980;302(20): 1109-1117. doi:10.1056/NEJM198005153022003 42. Kassirer JP, Moskowitz AJ, Lau J, Pauker SG. Decision analysis: a progress report. Ann Intern Med. 1987;106 (2):275-291. doi:10.7326/0003-4819-106-2-275 43. Rosenbloom ST, Chiu K-W, Byrne DW, Talbert DA, Neilson EG, Miller RA. Interventions to regulate ordering of serum magnesium levels: report of an unintended consequence of decision support. J Am Med Inform Assoc. 2005;12(5):546-553. doi:10.1197/jamia.M1811 JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 12/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests 44. Corson AH, Fan VS, White T, et al. A multifaceted hospitalist quality improvement intervention: decreased frequency of common labs. JHospMed. 2015;10(6):390-395. doi:10.1002/jhm.2354 45. Sadowski BW, Lane AB, Wood SM, Robinson SL, Kim CH. High-value, cost-conscious care: iterative systems- based interventions to reduce unnecessary laboratory testing. Am J Med. 2017;130(9):1112.e1-1112.e7. doi:10.1016/ j.amjmed.2017.02.029 46. Afshar N, Tabas J, Afshar K, Silbergleit R. Blood cultures for community-acquired pneumonia: are they worthy of two quality measures? a systematic review. JHospMed. 2009;4(2):112-123. doi:10.1002/jhm.382 47. Vukkadala N, Auerbach A. Clinical utility of routine CBC testing in patients with community-acquired pneumonia. JHospMed. 2017;12(5):336-338. doi:10.12788/jhm.2734 48. Last M. Online classification of nonstationary data streams. Intell Data Anal. 2002;6(2):129-147. doi:10.3233/ IDA-2002-6203 SUPPLEMENT. eFigure 1. Machine Learning Pipeline eFigure 2. ROC Curves for Stanford Standalone Labs eFigure 3. ROC Curves for Stanford Components eFigure 4. ROC Curves for UMich Standalone Labs eFigure 5. ROC Curves for UMich Components eFigure 6. ROC Curves for UCSF Standalone Labs eFigure 7. ROC Curves for UCSF Components eTable 1. Data Matrix Feature Summary eTable 2. Model Construction Summary eTable 3. Diagnostic Metrics for Top Stanford Standalone Labs eTable 4. Diagnostic Metrics for Common Stanford Components eTable 5. Diagnostic Metrics for Top UMich Standalone Labs eTable 6. Diagnostic Metrics for Common UMich Components eTable 7. Diagnostic Metrics for Top UCSF Standalone Labs eTable 8. Diagnostic Metrics for Common UCSF Components eTable 9. Diagnostic Metrics for Common Components in Transferability Study eTable 10. Medicare and Chargemaster Fees for Standalone Labs eTable 11. Top 3 Important Features for Top Stanford Standalone Labs eTable 12. Top 3 Important Features for Common Stanford Components eTable 13. Top 3 Important Features for Top UMich Standalone Labs eTable 14. Top 3 Important Features for Common UMich Components eTable 15. Top 3 Important Features for Top UCSF Standalone Labs eTable 16. Top 3 Important Features for Common UCSF Components eMethods. Technical Details of Machine Learning Algorithm JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 13/13 Supplementary Online Content Xu S, Hom J, Balasubramanian S, et al. Prevalence and predictability of low-yield inpatient laboratory diagnostic tests. JAMA Netw Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 eFigure 1. Machine Learning Pipeline eFigure 2. ROC Curves for Stanford Standalone Labs eFigure 3. ROC Curves for Stanford Components eFigure 4. ROC Curves for UMich Standalone Labs eFigure 5. ROC Curves for UMich Components eFigure 6. ROC Curves for UCSF Standalone Labs eFigure 7. ROC Curves for UCSF Components eTable 1. Data Matrix Feature Summary eTable 2. Model Construction Summary eTable 3. Diagnostic Metrics for Top Stanford Standalone Labs eTable 4. Diagnostic Metrics for Common Stanford Components eTable 5. Diagnostic Metrics for Top UMich Standalone Labs eTable 6. Diagnostic Metrics for Common UMich Components eTable 7. Diagnostic Metrics for Top UCSF Standalone Labs eTable 8. Diagnostic Metrics for Common UCSF Components eTable 9. Diagnostic Metrics for Common Components in Transferability Study eTable 10. Medicare and Chargemaster Fees for Standalone Labs eTable 11. Top 3 Important Features for Top Stanford Standalone Labs eTable 12. Top 3 Important Features for Common Stanford Components eTable 13. Top 3 Important Features for Top UMich Standalone Labs eTable 14. Top 3 Important Features for Common UMich Components eTable 15. Top 3 Important Features for Top UCSF Standalone Labs eTable 16. Top 3 Important Features for Common UCSF Components eMethods. Technical Details of Machine Learning Algorithm This supplementary material has been provided by the authors to give readers additional information about their work. © 2019 Xu S et al. JAMA Network Open. eFigure 1. Machine Learning Pipeline Data processing, machine learning, and statistical analysis pipeline. When applied to a different dataset, an extra data extraction step will need to be implemented that extracts raw data (labs, diagnoses, demographics, encounters, patient info, treatment teams, etc.) from their database into Stanford-like columns The prediction output is a list of predicted normality scores, which were then compared against actual order results (labels). ROC: receiver operating characteristic, AUROC (or C-statistic): Area Under the ROC curve, NPV: negative predictive value, PPV: positive predictive value. © 2019 Xu S et al. JAMA Network Open. © 2019 Xu S et al. JAMA Network Open. eFigure 2. ROC Curves for Stanford Standalone Labs eFigure 3. ROC Curves for Stanford Components © 2019 Xu S et al. JAMA Network Open. eFigure 4. ROC Curves for UMich Standalone Labs eFigure 5. ROC Curves for UMich Components © 2019 Xu S et al. JAMA Network Open. eFigure 6. ROC Curves for UCSF Standalone Labs eFigure 7. ROC Curves for UCSF Components © 2019 Xu S et al. JAMA Network Open. category type features Lab of binary Normalilty of the Most Recent Order Interest time Time of Order (Month), Time of Order (Hour) Features modeled as nominal value, sine, and cosine. time-binned Order History counts Events represent prior orders of lab of interest. Demograph integer Age ics binary Female, Male Asian, Black, Hispanic/Latino, Native American, Pacific Islander, White Hispanic/Latino, White Non- Hispanic/Latino, Other Race, Unknown Race Admission integer Days since Admission Treatment time-binned Cardiology, Cardiovascular ICU, Coronary Care Unit, Team counts Hematology/Oncology, Medical ICU, Medicine, Neurology, Psychiatry, Surgery, Surgical ICU, Transplant, Trauma Events represent patient being treated by specialty. Flow Sheet summary Diastolic Blood Pressure, FiO2, Glasgow Coma Scale statistics Score, Pulse, Respiration, Systolic Blood Pressure, Temperature, Urine Comorbiditi time-binned Cerebrovascular Disease, Chronic Obstructive es counts Pulmonary Disease (COPD), Congestive Heart Failure, Dementia, Diabetes, Diabetes Complications, Hemiplegia/Paraplegia, HIV/AIDS, Liver Damage (Mild), Liver Damage (Moderate to Severe), Malignancy, Metastatic Malignancy, Myocardial Infarction, Peptic Ulcer, Peripheral Vascular Disease, Renal Disease, Rheumatism Events represent comorbidity being added to problem list. Lab Results summary Albumin, Arterial CO2 Partial Pressure, Arterial O2 statistics Partial Pressure, Arterial pH, Blood Urea Nitrogen, C- reactive Protein, Calcium, CO2, Creatinine, Erythrocyte Sedimentation Rate, Hematocrit, Lactate, Platelet Count, Potassium, Sodium, Total Bilirubin, Troponin I, Venous CO2 Partial Pressure, Venous O2 Partial Pressure, Venous pH, White Blood Cell Count eTable 1. Data Matrix Feature Summary For each lab we studied, we constructed an M-by-N data matrix. Each of the N columns represents a single feature to potentially be used in the prediction of normal results for the lab of interest. In total, ~880 features were used for building each model (the precise value varied per lab test, based on the number of component results returned by the © 2019 Xu S et al. JAMA Network Open. clinical laboratory). The table shows the categories of features and their data types. Each of the time-binned counts represents an event aggregated at various time intervals (1, 2, 4, 7, 14, 30, 90, 180, 365, 730, and 1460 days before lab order) and days since last event. Each of the summary statistics represents a value aggregated over the past 3 (for vitals) or 14 days (for lab results) by count, normal count, minimum, maximum, median, standard deviation, first value, last value, slope, and days since first and last values. Algorithm Hyperparameters AdaBoost adaboost_algorithm: SAMME.R, base_estimator: decision-tree, class_weight: balanced, learning_rate: [0.001, 0.01, 0.1, 1.0, 10.0], n_estimators: [10, 20, 30, 40, 50] Decision Tree class_weight: balanced, criterion: gini, max_depth: [1, 2, 3, 4, 5, None], max_features: [sqrt, log2, None], max_leaf_nodes: None, min_impurity_decrease: 0.0, min_samples_leaf: [0.01, 0.1, 1.0, 10.0], min_samples_split: [0.02, 0.2, 2, 20], min_weight_fraction_leaf: 0.0, presort: None, splitter: best Gaussian Naive priors: [[0.0001, 0.9999], [0.001, 0.999], [0.01, 0.99], [0.05, 0.95], Bayes [0.1, 0.9], [0.25, 0.75], [0.5, 0.5], [0.75, 0.25], [0.9, 0.1], [0.95, 0.05], [0.99, 0.01], [0.999, 0.001], [0.9999, 0.0001]] L1 Logistic C: [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0], Regression class_weight: balanced, dual: False, fit_intercept: True, max_iter: 1024, penalty: L1, solver: SAGA, tol: 0.0001 Neural Network Activation: logistic, tanh, relu Layer_sizes: (880, 10, 1), (880, 10, 10, 10, 1), (880, 10, 10, 10, 10, 10, 1) Solver: lbgfs, sgd, adam Random Forest bootstrap: True, n_estimators: [2, 5, 10, 15, 20, 25], warm_start: False © 2019 Xu S et al. JAMA Network Open. Other hyperparameters same as Decision Tree. Regress and M: [1, 2, 3, 4, 5] Round Other hyperparameters same as L1 Logistic Regression. XGBoost Colsample_bytree: [0.6, 0.8, 1.0] Learning_rate: [0.001, 0.01, 0.1, 1.0, 10.0] Max_depth: [1, 2, 3, 4, 5] Min_child_weight: [1, 5, 10] Subsample: [0.6, 0.8, 1.0] eTable 2. Model Construction Summary Eight machine learning algorithms were applied to predict lab result. Each algorithm tune the performance of the algorithms. Hyperparameters with multiple values describe the hyperparameter search space, which was explored exhaustively and scored based on AUROC (C-statistics) through 10-fold cross validation. Lab Test Vol C 95% Prev Target NPV Sens Spec NPV Magnesium 4246 0.76 [0.74, 0.78] 26% 0.99 95% 29% 98% 17% 12% 0.60% 25% 62% Magnesium 4246 0.76 [0.74, 0.78] 26% 0.95 91% 36% 86% 47% 35% 3.6% 22% 39% Magnesium 4246 0.76 [0.74, 0.78] 26% 0.9 87% 43% 71% 67% 50% 7.6% 18% 24% Magnesium 4246 0.76 [0.73, 0.78] 26% 0.8 79% 63% 29% 94% 70% 18% 7.5% 4.4% Prothrombin Time 2244 0.89 [0.88, 0.91] 80% 0.99 92% 81% 100% 2.3% 0.45% 0.04% 80% 19% Prothrombin Time 2244 0.89 [0.88, 0.91] 80% 0.95 85% 81% 100% 3.6% 0.70% 0.12% 80% 19% Prothrombin Time 2244 0.89 [0.88, 0.91] 80% 0.9 82% 83% 99% 18% 3.4% 0.74% 80% 16% Prothrombin Time 2244 0.89 [0.88, 0.90] 80% 0.8 69% 87% 96% 39% 7.6% 3.4% 77% 12% Phosphorus 2120 0.74 [0.72, 0.76] 33% 0.9 84% 45% 78% 54% 36% 7.1% 26% 31% Phosphorus 2120 0.74 [0.72, 0.76] 33% 0.95 88% 39% 91% 30% 20% 2.8% 30% 47% Phosphorus 2120 0.74 [0.72, 0.76] 33% 0.8 77% 57% 52% 81% 55% 16% 17% 13% Phosphorus 2120 0.74 [0.72, 0.76] 33% 0.99 88% 34% 98% 6.7% 4.5% 0.63% 32% 63% Partial Thromboplastin 1471 0.86 [0.85, 0.87] 61% 0.99 90% 62% 100% 4.1% 1.6% 0.18% 61% 37% Time Partial Thromboplastin 1471 0.86 [0.84, 0.87] 61% 0.9 81% 69% 96% 30% 12% 2.8% 59% 27% Time Partial Thromboplastin 1471 0.86 [0.85, 0.87] 61% 0.8 74% 78% 87% 61% 23% 8.2% 53% 15% Time Partial Thromboplastin 1471 0.86 [0.85, 0.87] 61% 0.95 87% 65% 98% 17% 6.5% 1.0% 60% 32% Time Lactate 1230 0.87 [0.85, 0.88] 29% 0.8 77% 90% 28% 99% 71% 21% 8.0% 0.86% © 2019 Xu S et al. JAMA Network Open. Lactate 1230 0.87 [0.85, 0.88] 29% 0.95 91% 56% 82% 74% 53% 5.2% 23% 19% Lactate 1230 0.87 [0.85, 0.89] 29% 0.99 97% 35% 98% 28% 20% 0.54% 28% 51% Lactate 1230 0.87 [0.85, 0.89] 29% 0.9 87% 72% 65% 90% 64% 9.9% 19% 7.4% Calcium Ionized 1197 0.72 [0.70, 0.74] 61% 0.95 90% 62% 100% 4.8% 1.9% 0.21% 61% 37% Calcium Ionized 1197 0.72 [0.69, 0.74] 61% 0.9 82% 63% 99% 9.0% 3.5% 0.80% 60% 36% Calcium Ionized 1197 0.72 [0.70, 0.74] 61% 0.8 71% 66% 93% 25% 10% 4.1% 57% 29% Calcium Ionized 1197 0.72 [0.70, 0.74] 61% 0.99 96% 61% 100% 2.4% 0.93% 0.04% 61% 38% Potassium 752 0.81 [0.79, 0.84] 12% 0.95 92% 40% 43% 91% 80% 7.0% 5.2% 7.9% Potassium 752 0.81 [0.79, 0.84] 12% 0.8 88% - 0 100% 88% 12% 0 0 Potassium 752 0.81 [0.79, 0.84] 12% 0.9 88% 64% 2.8% 100% 88% 12% 0.34% 0.19% Potassium 752 0.81 [0.79, 0.84] 12% 0.99 97% 21% 89% 54% 48% 1.3% 11% 40% Troponin I 534 0.92 [0.91, 0.93] 33% 0.99 95% 38% 98% 23% 16% 0.79% 32% 52% Troponin I 534 0.92 [0.91, 0.93] 33% 0.95 93% 67% 88% 79% 53% 4.0% 29% 14% Troponin I 534 0.92 [0.91, 0.93] 33% 0.9 89% 88% 76% 95% 64% 7.8% 25% 3.3% Troponin I 534 0.92 [0.91, 0.93] 33% 0.8 79% 99% 45% 100% 67% 18% 15% 0.16% LDH Total 455 0.93 [0.93, 0.94] 47% 0.8 79% 90% 72% 93% 50% 13% 34% 3.8% LDH Total 455 0.93 [0.93, 0.94] 47% 0.9 90% 78% 90% 78% 42% 4.6% 42% 12% LDH Total 455 0.93 [0.93, 0.94] 47% 0.99 98% 60% 99% 43% 23% 0.35% 46% 30% LDH Total 455 0.93 [0.93, 0.94] 47% 0.95 95% 71% 96% 65% 35% 1.8% 45% 18% Heparin 423 0.76 [0.74, 0.78] 63% 0.99 0 63% 100% 0 0 0.04% 63% 37% Heparin 423 0.76 [0.74, 0.78] 63% 0.8 72% 71% 92% 37% 14% 5.2% 58% 24% Heparin 423 0.76 [0.74, 0.78] 63% 0.9 74% 68% 95% 24% 8.9% 3.2% 60% 28% Heparin 423 0.76 [0.74, 0.78] 63% 0.95 0 63% 100% 0 0 0.04% 63% 37% Urinalysis 417 0.71 [0.63, 0.80] 80% 0.95 - 80% 100% 0 0 0 80% 20% Urinalysis 417 0.71 [0.63, 0.79] 80% 0.8 - 80% 100% 0 0 0 80% 20% Urinalysis 417 0.71 [0.63, 0.79] 80% 0.99 - 80% 100% 0 0 0 80% 20% Urinalysis 417 0.71 [0.63, 0.79] 80% 0.9 - 80% 100% 0 0 0 80% 20% Blood Culture (Aerobic & 400 0.66 [0.61, 0.71] 8.1% 0.8 92% - 0 100% 92% 8.1% 0 0 Anaerobic) Blood Culture (Aerobic & 400 0.66 [0.61, 0.71] 8.1% 0.9 92% - 0 100% 92% 8.1% 0 0 Anaerobic) Blood Culture (Aerobic & 400 0.66 [0.61, 0.71] 8.1% 0.95 93% 16% 16% 93% 85% 6.8% 1.3% 6.6% Anaerobic) Blood Culture (Aerobic & 400 0.66 [0.61, 0.71] 8.1% 0.99 94% 14% 45% 76% 70% 4.5% 3.6% 22% Anaerobic) Blood Culture (2 371 0.62 [0.58, 0.67] 9.1% 0.95 93% 12% 61% 54% 49% 3.6% 5.6% 42% Aerobic) Blood Culture (2 371 0.62 [0.58, 0.67] 9.1% 0.8 91% - 0 100% 91% 9.1% 0 0 Aerobic) Blood Culture (2 371 0.62 [0.58, 0.67] 9.1% 0.9 91% - 0 100% 91% 9.1% 0 0 Aerobic) © 2019 Xu S et al. JAMA Network Open. Blood Culture (2 371 0.62 [0.57, 0.67] 9.1% 0.99 - 9.1% 100% 0 0 0 9.1% 91% Aerobic) Sodium 361 0.92 [0.91, 0.93] 57% 0.99 100% 58% 100% 3.1% 1.4% 0 57% 42% Sodium 361 0.92 [0.91, 0.93] 57% 0.95 93% 66% 98% 35% 15% 1.1% 56% 28% Sodium 361 0.92 [0.91, 0.93] 57% 0.8 76% 91% 78% 90% 39% 12% 45% 4.5% Sodium 361 0.92 [0.91, 0.93] 57% 0.9 87% 79% 92% 68% 29% 4.5% 52% 14% Lidocaine 315 0.83 [0.79, 0.86] 23% 0.99 94% 39% 89% 57% 44% 2.6% 21% 33% Lidocaine 315 0.83 [0.79, 0.86] 23% 0.95 91% 55% 73% 82% 63% 6.3% 17% 14% Lidocaine 315 0.83 [0.79, 0.86] 23% 0.8 77% 100% 0.62% 100% 77% 23% 0.14% 0 Lidocaine 315 0.83 [0.79, 0.86] 23% 0.9 83% 61% 35% 93% 71% 15% 8.2% 5.3% Hematocrit 288 0.9 [0.88, 0.92] 93% 0.99 100% 93% 100% 1.0% 0.08% 0 93% 7.2% Hematocrit 288 0.9 [0.87, 0.92] 93% 0.8 50% 94% 99% 16% 1.1% 1.1% 92% 6.1% Hematocrit 288 0.9 [0.88, 0.92] 93% 0.9 44% 93% 99% 7.9% 0.57% 0.72% 92% 6.7% Hematocrit 288 0.9 [0.88, 0.92] 93% 0.95 58% 93% 100% 3.7% 0.27% 0.19% 93% 7.0% Urine Culture 257 0.71 [0.68, 0.74] 36% 0.99 100% 36% 100% 2.2% 1.4% 0 36% 63% Urine Culture 257 0.71 [0.68, 0.74] 36% 0.95 90% 41% 95% 24% 15% 1.7% 34% 49% Urine Culture 257 0.71 [0.68, 0.74] 36% 0.9 84% 44% 86% 40% 26% 5.0% 31% 39% Urine Culture 257 0.71 [0.68, 0.74] 36% 0.8 75% 52% 58% 71% 45% 15% 21% 19% Urinalysis With 246 0.63 [0.57, 0.70] 72% 0.9 40% 75% 86% 24% 6.6% 10% 62% 21% Microscopic Urinalysis With 246 0.63 [0.56, 0.70] 72% 0.8 35% 75% 78% 32% 8.8% 16% 56% 19% Microscopic Urinalysis With 246 0.63 [0.57, 0.70] 72% 0.95 48% 75% 93% 17% 4.7% 5.0% 67% 23% Microscopic Urinalysis With 246 0.63 [0.57, 0.70] 72% 0.99 43% 73% 97% 6.8% 1.9% 2.5% 70% 26% Microscopic Uric Acid 229 0.97 [0.96, 0.98] 3.8% 0.99 98% 51% 50% 98% 94% 1.9% 1.9% 1.8% Uric Acid 229 0.97 [0.96, 0.98] 3.8% 0.95 96% - 0 100% 96% 3.8% 0 0 Uric Acid 229 0.97 [0.96, 0.98] 3.8% 0.8 96% - 0 100% 96% 3.8% 0 0 Uric Acid 229 0.97 [0.96, 0.98] 3.8% 0.9 96% - 0 100% 96% 3.8% 0 0 Hemoglobin A1c 225 0.81 [0.79, 0.82] 59% 0.95 87% 63% 99% 14% 5.8% 0.86% 58% 35% Hemoglobin A1c 225 0.81 [0.79, 0.82] 59% 0.99 93% 61% 100% 6.8% 2.8% 0.20% 59% 38% Hemoglobin A1c 225 0.81 [0.79, 0.82] 59% 0.9 85% 66% 97% 28% 11% 2.0% 57% 29% Hemoglobin A1c 225 0.81 [0.79, 0.82] 59% 0.8 73% 74% 86% 55% 23% 8.4% 51% 18% Sepsis Protocol Lactate 187 0.86 [0.84, 0.88] 17% 0.99 97% 28% 92% 52% 44% 1.3% 15% 40% Sepsis Protocol Lactate 187 0.86 [0.84, 0.88] 17% 0.8 83% - 0 100% 83% 17% 0 0 Sepsis Protocol Lactate 187 0.86 [0.84, 0.88] 17% 0.9 87% 80% 30% 98% 82% 12% 5.0% 1.3% Sepsis Protocol Lactate 187 0.86 [0.84, 0.88] 17% 0.95 92% 55% 60% 90% 75% 6.7% 10% 8.4% iSTAT Troponin I 184 0.79 [0.75, 0.82] 13% 0.99 95% 16% 91% 26% 23% 1.1% 12% 64% iSTAT Troponin I 184 0.79 [0.75, 0.82] 13% 0.95 93% 55% 54% 93% 81% 6.2% 7.1% 5.9% iSTAT Troponin I 184 0.79 [0.75, 0.82] 13% 0.9 87% 93% 4.6% 100% 87% 13% 0.62% 0.05% © 2019 Xu S et al. JAMA Network Open. iSTAT Troponin I 184 0.79 [0.75, 0.82] 13% 0.8 87% - 0 100% 87% 13% 0 0 Platelet Count 174 0.95 [0.94, 0.96] 44% 0.99 98% 57% 99% 41% 23% 0.44% 43% 33% Platelet Count 174 0.95 [0.94, 0.96] 44% 0.95 92% 76% 92% 78% 44% 3.6% 40% 12% Platelet Count 174 0.95 [0.94, 0.96] 44% 0.9 89% 90% 85% 92% 52% 6.6% 37% 4.2% Platelet Count 174 0.95 [0.94, 0.96] 44% 0.8 80% 98% 68% 99% 55% 14% 30% 0.67% Lipase 174 0.79 [0.77, 0.82] 23% 0.99 93% 26% 96% 17% 13% 0.98% 22% 64% Lipase 174 0.79 [0.77, 0.81] 23% 0.95 91% 35% 82% 53% 41% 4.3% 19% 36% Lipase 174 0.79 [0.77, 0.82] 23% 0.8 80% 92% 15% 100% 76% 20% 3.6% 0.30% Lipase 174 0.79 [0.77, 0.82] 23% 0.9 88% 55% 61% 85% 65% 9.1% 14% 11% Procalcitonin 174 0.89 [0.88, 0.90] 52% 0.95 91% 61% 97% 34% 17% 1.6% 50% 32% Procalcitonin 174 0.89 [0.88, 0.90] 52% 0.9 85% 73% 89% 65% 31% 5.5% 46% 17% Procalcitonin 174 0.89 [0.88, 0.90] 52% 0.8 77% 86% 75% 87% 42% 13% 39% 6.2% Procalcitonin 174 0.89 [0.88, 0.90] 52% 0.99 96% 53% 100% 5.7% 2.8% 0.12% 52% 46% Lactic Acid 152 0.87 [0.85, 0.88] 25% 0.9 87% 72% 59% 92% 69% 10% 15% 5.8% Lactic Acid 152 0.87 [0.85, 0.88] 25% 0.8 79% 95% 20% 100% 75% 20% 5.0% 0.25% Lactic Acid 152 0.87 [0.85, 0.88] 25% 0.95 92% 54% 79% 78% 58% 5.3% 20% 17% Lactic Acid 152 0.87 [0.85, 0.88] 25% 0.99 97% 34% 97% 36% 27% 0.85% 24% 48% Fibrinogen 148 0.78 [0.76, 0.80] 61% 0.99 100% 61% 100% 0.41% 0.16% 0 61% 39% Fibrinogen 148 0.78 [0.76, 0.80] 61% 0.95 79% 63% 98% 11% 4.1% 1.1% 60% 35% Fibrinogen 148 0.78 [0.77, 0.80] 61% 0.9 76% 67% 95% 27% 10% 3.3% 58% 29% Fibrinogen 148 0.78 [0.76, 0.80] 61% 0.8 74% 72% 90% 44% 17% 6.1% 55% 22% Thyroid Stimulating 145 0.64 [0.62, 0.67] 27% 0.9 81% 34% 67% 52% 38% 9.0% 18% 35% Hormone Thyroid Stimulating 145 0.64 [0.62, 0.67] 27% 0.8 77% 54% 26% 92% 67% 20% 7.1% 6.0% Hormone Thyroid Stimulating 145 0.64 [0.62, 0.67] 27% 0.99 84% 27% 100% 0.87% 0.63% 0.12% 27% 72% Hormone Thyroid Stimulating 145 0.64 [0.62, 0.67] 27% 0.95 83% 30% 87% 24% 18% 3.7% 24% 55% Hormone Creatine Kinase 126 0.94 [0.93, 0.95] 48% 0.9 87% 88% 86% 89% 47% 6.8% 41% 5.6% Creatine Kinase 126 0.94 [0.93, 0.95] 48% 0.8 80% 96% 73% 97% 51% 13% 35% 1.4% Creatine Kinase 126 0.94 [0.93, 0.95] 48% 0.99 95% 66% 97% 55% 29% 1.5% 46% 24% Creatine Kinase 126 0.94 [0.93, 0.95] 48% 0.95 91% 79% 92% 78% 41% 4.0% 44% 12% C-Reactive Protein 125 0.86 [0.84, 0.88] 87% 0.99 82% 87% 100% 3.9% 0.52% 0.11% 87% 13% C-Reactive Protein 125 0.86 [0.84, 0.88] 87% 0.8 68% 89% 98% 22% 3.0% 1.4% 85% 10% C-Reactive Protein 125 0.86 [0.84, 0.88] 87% 0.95 81% 87% 100% 6.1% 0.81% 0.18% 86% 13% C-Reactive Protein 125 0.86 [0.84, 0.88] 87% 0.9 72% 88% 99% 12% 1.6% 0.59% 86% 12% -proBNP 122 0.81 [0.79, 0.83] 82% 0.8 74% 84% 99% 17% 3.2% 1.1% 81% 15% -proBNP 122 0.81 [0.79, 0.83] 82% 0.9 76% 83% 99% 9.4% 1.7% 0.53% 81% 17% -proBNP 122 0.81 [0.79, 0.83] 82% 0.95 70% 82% 100% 1.6% 0.29% 0.12% 82% 18% -proBNP 122 0.81 [0.79, 0.83] 82% 0.99 71% 82% 100% 1.1% 0.20% 0.08% 82% 18% © 2019 Xu S et al. JAMA Network Open. Triglycerides 105 0.86 [0.84, 0.87] 42% 0.8 76% 84% 61% 91% 53% 17% 26% 5.0% Triglycerides 105 0.86 [0.84, 0.87] 42% 0.9 84% 66% 82% 69% 40% 7.5% 35% 18% Triglycerides 105 0.86 [0.84, 0.87] 42% 0.95 92% 54% 95% 41% 24% 2.1% 40% 34% Triglycerides 105 0.86 [0.84, 0.87] 42% 0.99 95% 47% 99% 19% 11% 0.55% 42% 46% CK, MB 96 0.9 [0.88, 0.91] 49% 0.8 79% 86% 75% 88% 45% 12% 37% 5.9% CK, MB 96 0.9 [0.88, 0.91] 49% 0.9 86% 74% 88% 70% 36% 5.7% 43% 15% CK, MB 96 0.9 [0.88, 0.91] 49% 0.95 91% 64% 95% 48% 24% 2.4% 46% 27% CK, MB 96 0.9 [0.88, 0.91] 49% 0.99 96% 53% 99% 15% 7.7% 0.35% 49% 43% C.diff Toxin B Gene 95 0.65 [0.62, 0.68] 18% 0.8 82% - 0 100% 82% 18% 0 0 C.diff Toxin B Gene 95 0.65 [0.62, 0.68] 18% 0.9 87% 27% 55% 67% 55% 8.2% 9.9% 27% C.diff Toxin B Gene 95 0.65 [0.62, 0.68] 18% 0.95 89% 21% 85% 28% 23% 2.7% 15% 59% C.diff Toxin B Gene 95 0.65 [0.62, 0.68] 18% 0.99 80% 18% 99% 1.2% 0.96% 0.24% 18% 81% Osmolality 88 0.92 [0.91, 0.93] 53% 0.8 76% 91% 75% 91% 43% 13% 40% 4.0% Osmolality 88 0.92 [0.91, 0.93] 53% 0.9 88% 82% 91% 77% 36% 4.9% 48% 11% Osmolality 88 0.92 [0.91, 0.93] 53% 0.95 94% 70% 97% 53% 25% 1.7% 52% 22% Osmolality 88 0.92 [0.91, 0.93] 53% 0.99 96% 56% 100% 11% 5.4% 0.22% 53% 41% Respiratory Culture And 87 0.62 [0.58, 0.65] 35% 0.99 80% 35% 100% 0.59% 0.38% 0.10% 34% 65% Gram Stain Respiratory Culture And 87 0.62 [0.58, 0.65] 35% 0.95 92% 35% 100% 1.8% 1.2% 0.10% 34% 64% Gram Stain Respiratory Culture And 87 0.62 [0.58, 0.65] 35% 0.9 78% 36% 92% 15% 9.5% 2.7% 32% 56% Gram Stain Respiratory Culture And 87 0.62 [0.58, 0.65] 35% 0.8 72% 42% 55% 61% 40% 15% 19% 26% Gram Stain Sedimentation Rate 85 0.83 [0.81, 0.85] 69% 0.99 100% 70% 100% 2.0% 0.62% 0 69% 30% (ESR) Sedimentation Rate 85 0.83 [0.81, 0.84] 69% 0.95 86% 73% 99% 19% 5.9% 1.0% 68% 25% (ESR) Sedimentation Rate 85 0.83 [0.81, 0.84] 69% 0.9 80% 75% 97% 30% 9.1% 2.3% 67% 22% (ESR) Sedimentation Rate 85 0.83 [0.81, 0.85] 69% 0.8 70% 78% 92% 41% 13% 5.4% 64% 18% (ESR) Ferritin 82 0.81 [0.79, 0.82] 62% 0.95 91% 63% 100% 4.4% 1.7% 0.16% 62% 36% Ferritin 82 0.81 [0.79, 0.82] 62% 0.8 73% 72% 91% 42% 16% 5.8% 57% 22% Ferritin 82 0.81 [0.79, 0.82] 62% 0.9 80% 66% 97% 17% 6.6% 1.6% 61% 31% Ferritin 82 0.81 [0.79, 0.82] 62% 0.99 93% 63% 100% 2.6% 0.98% 0.08% 62% 37% Albumin 78 0.9 [0.88, 0.91] 89% 0.95 75% 90% 99% 15% 1.7% 0.57% 88% 9.7% Albumin 78 0.9 [0.88, 0.91] 89% 0.9 71% 91% 99% 26% 2.9% 1.2% 87% 8.5% Albumin 78 0.9 [0.88, 0.91] 89% 0.8 60% 92% 97% 35% 4.0% 2.6% 86% 7.4% Albumin 78 0.9 [0.87, 0.91] 89% 0.99 74% 89% 100% 5.0% 0.57% 0.20% 88% 11% Ammonia 77 0.78 [0.76, 0.79] 59% 0.8 73% 69% 90% 39% 16% 5.9% 54% 25% Ammonia 77 0.78 [0.76, 0.79] 59% 0.9 78% 63% 97% 17% 6.7% 1.9% 58% 34% © 2019 Xu S et al. JAMA Network Open. Ammonia 77 0.78 [0.76, 0.80] 59% 0.95 80% 61% 99% 8.3% 3.4% 0.85% 59% 37% Ammonia 77 0.78 [0.76, 0.79] 59% 0.99 77% 60% 99% 3.4% 1.4% 0.42% 59% 39% Specific Gravity 76 0.67 [0.54, 0.79] 0.85% 0.99 99% - 0 100% 99% 0.85% 0 0 Specific Gravity 76 0.67 [0.55, 0.79] 0.85% 0.95 99% - 0 100% 99% 0.85% 0 0 Specific Gravity 76 0.67 [0.55, 0.78] 0.85% 0.9 99% - 0 100% 99% 0.85% 0 0 Specific Gravity 76 0.67 [0.54, 0.79] 0.85% 0.8 99% - 0 100% 99% 0.85% 0 0 Fungal Culture 74 0.76 [0.73, 0.79] 18% 0.8 82% - 0 100% 82% 18% 0 0 Fungal Culture 74 0.76 [0.73, 0.80] 18% 0.9 93% 28% 81% 54% 45% 3.5% 15% 37% Fungal Culture 74 0.76 [0.72, 0.80] 18% 0.95 - 18% 100% 0 0 0 18% 82% Fungal Culture 74 0.76 [0.73, 0.80] 18% 0.99 - 18% 100% 0 0 0 18% 82% Haptoglobin 71 0.77 [0.75, 0.79] 43% 0.8 74% 65% 66% 73% 42% 14% 28% 15% Haptoglobin 71 0.77 [0.75, 0.79] 43% 0.9 83% 53% 89% 40% 23% 4.8% 38% 34% Haptoglobin 71 0.77 [0.75, 0.79] 43% 0.95 88% 47% 96% 20% 11% 1.6% 41% 46% Haptoglobin 71 0.77 [0.75, 0.79] 43% 0.99 97% 44% 100% 5.5% 3.2% 0.09% 43% 54% Anaerobic Culture 69 0.78 [0.75, 0.82] 13% 0.9 90% 51% 25% 97% 84% 9.5% 3.2% 3.0% Anaerobic Culture 69 0.78 [0.75, 0.82] 13% 0.99 99% 17% 97% 33% 29% 0.41% 12% 58% Anaerobic Culture 69 0.78 [0.75, 0.82] 13% 0.95 94% 28% 64% 76% 66% 4.6% 8.1% 21% Anaerobic Culture 69 0.78 [0.75, 0.82] 13% 0.8 87% - 0 100% 87% 13% 0 0 Fluid Culture And Gram 69 0.73 [0.69, 0.76] 26% 0.99 91% 29% 94% 20% 15% 1.5% 25% 59% Stain Fluid Culture And Gram 69 0.73 [0.69, 0.76] 26% 0.95 85% 41% 67% 66% 49% 8.7% 17% 25% Stain Fluid Culture And Gram 69 0.73 [0.69, 0.76] 26% 0.9 80% 50% 42% 85% 63% 15% 11% 11% Stain Fluid Culture And Gram 69 0.73 [0.69, 0.77] 26% 0.8 76% 67% 10% 98% 73% 23% 2.7% 1.4% Stain Cmv Dna Pcr Quant 66 0.79 [0.76, 0.82] 27% 0.8 79% 90% 28% 99% 72% 20% 7.7% 0.85% Cmv Dna Pcr Quant 66 0.79 [0.76, 0.82] 27% 0.9 85% 61% 58% 86% 63% 11% 16% 10% Cmv Dna Pcr Quant 66 0.79 [0.76, 0.82] 27% 0.95 88% 44% 76% 63% 46% 6.5% 21% 27% Cmv Dna Pcr Quant 66 0.79 [0.76, 0.82] 27% 0.99 92% 31% 95% 21% 15% 1.3% 26% 57% Blood Cult Central Line 62 0.65 [0.60, 0.69] 11% 0.95 90% 20% 19% 90% 80% 8.9% 2.1% 8.5% Catheter By Nurse Blood Cult Central Line 62 0.65 [0.60, 0.70] 11% 0.8 89% - 0 100% 89% 11% 0 0 Catheter By Nurse Blood Cult Central Line 62 0.65 [0.60, 0.70] 11% 0.9 89% 100% 2.1% 100% 89% 11% 0.23% 0 Catheter By Nurse Blood Cult Central Line 62 0.65 [0.61, 0.69] 11% 0.99 93% 15% 68% 52% 46% 3.6% 7.4% 43% Catheter By Nurse T4, FREE 61 0.68 [0.64, 0.71] 11% 0.9 89% 100% 0.34% 100% 89% 11% 0.04% 0 T4, FREE 61 0.68 [0.64, 0.71] 11% 0.95 92% 28% 37% 88% 78% 7.1% 4.2% 11% T4, FREE 61 0.68 [0.64, 0.71] 11% 0.99 95% 13% 90% 24% 21% 1.1% 10% 68% T4, FREE 61 0.68 [0.64, 0.71] 11% 0.8 89% - 0 100% 89% 11% 0 0 © 2019 Xu S et al. JAMA Network Open. Transferrin Saturation 60 0.65 [0.60, 0.70] 88% 0.8 - 88% 100% 0 0 0 88% 12% Transferrin Saturation 60 0.65 [0.60, 0.70] 88% 0.99 - 88% 100% 0 0 0 88% 12% Transferrin Saturation 60 0.65 [0.60, 0.70] 88% 0.9 - 88% 100% 0 0 0 88% 12% Transferrin Saturation 60 0.65 [0.60, 0.70] 88% 0.95 - 88% 100% 0 0 0 88% 12% Vitamin B12 60 0.72 [0.70, 0.74] 32% 0.8 77% 54% 50% 80% 54% 16% 16% 14% Vitamin B12 60 0.72 [0.70, 0.74] 32% 0.9 85% 40% 86% 39% 26% 4.5% 28% 42% Vitamin B12 60 0.72 [0.70, 0.74] 32% 0.95 87% 35% 96% 13% 8.8% 1.3% 31% 59% Vitamin B12 60 0.72 [0.70, 0.74] 32% 0.99 93% 32% 100% 0.82% 0.56% 0.04% 32% 67% Blood Cult - First Set 60 0.67 [0.62, 0.71] 11% 0.99 92% 18% 54% 70% 62% 5.0% 6.0% 27% Blood Cult - First Set 60 0.67 [0.62, 0.71] 11% 0.9 89% 0 0 100% 89% 11% 0 0.08% Blood Cult - First Set 60 0.67 [0.62, 0.72] 11% 0.8 89% - 0 100% 89% 11% 0 0 Blood Cult - First Set 60 0.67 [0.62, 0.71] 11% 0.95 90% 25% 16% 94% 84% 9.3% 1.8% 5.2% Prealbumin 59 0.85 [0.83, 0.86] 77% 0.99 77% 77% 100% 1.9% 0.45% 0.14% 77% 23% Prealbumin 59 0.85 [0.83, 0.86] 77% 0.95 90% 79% 100% 13% 3.0% 0.32% 76% 20% Prealbumin 59 0.85 [0.83, 0.86] 77% 0.9 85% 81% 99% 23% 5.4% 0.95% 76% 18% Prealbumin 59 0.85 [0.83, 0.86] 77% 0.8 79% 84% 97% 39% 9.0% 2.4% 74% 14% Osmolality, Urine 58 0.92 [0.90, 0.93] 7.9% 0.95 94% 50% 22% 98% 90% 6.1% 1.8% 1.8% Osmolality, Urine 58 0.92 [0.90, 0.93] 7.9% 0.99 97% 34% 70% 88% 81% 2.4% 5.5% 11% Osmolality, Urine 58 0.92 [0.90, 0.93] 7.9% 0.9 92% - 0 100% 92% 7.9% 0 0 Osmolality, Urine 58 0.92 [0.90, 0.93] 7.9% 0.8 92% - 0 100% 92% 7.9% 0 0 Digoxin 55 0.8 [0.77, 0.83] 22% 0.99 - 22% 100% 0 0 0 22% 78% Digoxin 55 0.8 [0.77, 0.83] 22% 0.95 95% 36% 89% 56% 44% 2.4% 19% 35% Digoxin 55 0.8 [0.77, 0.83] 22% 0.9 92% 44% 78% 72% 57% 4.7% 17% 22% Digoxin 55 0.8 [0.77, 0.83] 22% 0.8 81% 56% 19% 96% 75% 17% 4.1% 3.2% Reticulocyte Count 50 0.79 [0.77, 0.81] 63% 0.9 81% 70% 96% 28% 10% 2.5% 61% 27% Automated Reticulocyte Count 50 0.79 [0.77, 0.81] 63% 0.99 81% 64% 100% 3.3% 1.2% 0.29% 63% 36% Automated Reticulocyte Count 50 0.79 [0.77, 0.81] 63% 0.8 71% 73% 89% 45% 16% 6.9% 56% 20% Automated Reticulocyte Count 50 0.79 [0.77, 0.81] 63% 0.95 85% 66% 99% 11% 4.2% 0.74% 62% 33% Automated Cortisol 48 0.71 [0.68, 0.73] 24% 0.8 77% 78% 6.5% 99% 75% 23% 1.6% 0.45% Cortisol 48 0.71 [0.69, 0.73] 24% 0.9 83% 38% 52% 73% 56% 12% 12% 20% Cortisol 48 0.71 [0.69, 0.73] 24% 0.95 88% 33% 78% 50% 38% 5.3% 19% 38% Cortisol 48 0.71 [0.69, 0.73] 24% 0.99 96% 29% 97% 23% 18% 0.76% 23% 58% Hepatitis B Antigen 45 0.8 [0.73, 0.86] 2.9% 0.95 97% - 0 100% 97% 2.9% 0 0 Hepatitis B Antigen 45 0.8 [0.73, 0.86] 2.9% 0.9 97% - 0 100% 97% 2.9% 0 0 Hepatitis B Antigen 45 0.8 [0.73, 0.86] 2.9% 0.99 98% 60% 22% 100% 97% 2.3% 0.65% 0.43% Hepatitis B Antigen 45 0.8 [0.73, 0.86] 2.9% 0.8 97% - 0 100% 97% 2.9% 0 0 Iron 40 0.74 [0.71, 0.77] 40% 0.99 91% 44% 98% 17% 10% 0.95% 39% 50% © 2019 Xu S et al. JAMA Network Open. Iron 40 0.74 [0.71, 0.78] 40% 0.8 73% 64% 57% 78% 47% 17% 23% 13% Iron 40 0.74 [0.71, 0.77] 40% 0.9 78% 53% 77% 53% 31% 9.1% 31% 28% Iron 40 0.74 [0.71, 0.78] 40% 0.95 81% 48% 88% 36% 22% 5.0% 35% 38% AFB Culture 40 0.68 [0.59, 0.76] 4.9% 0.99 95% 0 0 100% 95% 4.9% 0 0.10% AFB Culture 40 0.68 [0.59, 0.76] 4.9% 0.8 95% - 0 100% 95% 4.9% 0 0 AFB Culture 40 0.68 [0.59, 0.77] 4.9% 0.95 95% 0 0 100% 95% 4.9% 0 0.10% AFB Culture 40 0.68 [0.60, 0.76] 4.9% 0.9 95% - 0 100% 95% 4.9% 0 0 Calcium 39 0.86 [0.84, 0.87] 66% 0.99 89% 67% 100% 4.1% 1.4% 0.17% 66% 32% Calcium 39 0.86 [0.84, 0.87] 66% 0.9 82% 75% 96% 36% 12% 2.7% 64% 21% Calcium 39 0.86 [0.84, 0.87] 66% 0.8 75% 80% 91% 55% 18% 6.3% 60% 15% Calcium 39 0.86 [0.84, 0.87] 66% 0.95 86% 70% 99% 16% 5.5% 0.93% 65% 28% Biopsy/tissue With Gram 36 0.75 [0.72, 0.79] 37% 0.8 72% 62% 42% 85% 54% 21% 16% 9.5% Stain Biopsy/tissue With Gram 36 0.75 [0.72, 0.79] 37% 0.9 82% 54% 76% 63% 40% 8.7% 28% 24% Stain Biopsy/tissue With Gram 36 0.75 [0.72, 0.79] 37% 0.99 88% 38% 99% 5.9% 3.8% 0.54% 36% 60% Stain Biopsy/tissue With Gram 36 0.75 [0.71, 0.79] 37% 0.95 89% 44% 93% 32% 20% 2.4% 34% 43% Stain Afb Culture 35 0.81 [0.68, 0.92] 1.5% 0.8 99% - 0 100% 99% 1.5% 0 0 Afb Culture 35 0.81 [0.66, 0.92] 1.5% 0.9 99% - 0 100% 99% 1.5% 0 0 Afb Culture 35 0.81 [0.66, 0.92] 1.5% 0.95 99% - 0 100% 99% 1.5% 0 0 Afb Culture 35 0.81 [0.65, 0.92] 1.5% 0.99 100% 2.4% 92% 45% 44% 0.12% 1.3% 54% Pregnancy Test 34 0.64 [0.54, 0.74] 1.8% 0.95 98% - 0 100% 98% 1.8% 0 0 Pregnancy Test 34 0.64 [0.54, 0.73] 1.8% 0.9 98% - 0 100% 98% 1.8% 0 0 Pregnancy Test 34 0.64 [0.54, 0.74] 1.8% 0.8 98% - 0 100% 98% 1.8% 0 0 Pregnancy Test 34 0.64 [0.54, 0.73] 1.8% 0.99 - 1.8% 100% 0 0 0 1.8% 98% Urea Nitrogen 34 0.91 [0.89, 0.92] 51% 0.8 77% 89% 74% 90% 45% 13% 38% 4.7% Urea Nitrogen 34 0.91 [0.89, 0.92] 51% 0.9 87% 82% 88% 81% 40% 6.2% 44% 9.5% Urea Nitrogen 34 0.91 [0.89, 0.92] 51% 0.95 90% 75% 93% 68% 33% 3.7% 47% 16% Urea Nitrogen 34 0.91 [0.89, 0.92] 51% 0.99 96% 54% 99% 13% 6.3% 0.27% 50% 43% Gram Stain 31 0.57 [0.52, 0.63] 11% 0.9 - 11% 100% 0 0 0 11% 89% Gram Stain 31 0.57 [0.52, 0.63] 11% 0.99 - 11% 100% 0 0 0 11% 89% Gram Stain 31 0.57 [0.52, 0.63] 11% 0.95 - 11% 100% 0 0 0 11% 89% Gram Stain 31 0.57 [0.52, 0.63] 11% 0.8 - 11% 100% 0 0 0 11% 89% 29 0.66 [0.61, 0.69] 54% 0.8 64% 54% 99% 3.1% 1.4% 0.80% 53% 45% 29 0.66 [0.61, 0.70] 54% 0.9 78% 54% 99% 2.4% 1.1% 0.32% 54% 45% 29 0.66 [0.61, 0.70] 54% 0.95 80% 54% 100% 1.4% 0.64% 0.16% 54% 45% 29 0.66 [0.61, 0.70] 54% 0.99 80% 54% 100% 1.4% 0.64% 0.16% 54% 45% Protein Total 27 0.76 [0.73, 0.78] 67% 0.99 81% 67% 100% 2.6% 0.88% 0.20% 66% 33% Protein Total 27 0.76 [0.73, 0.78] 67% 0.95 83% 68% 99% 8.1% 2.7% 0.54% 66% 31% © 2019 Xu S et al. JAMA Network Open. Protein Total 27 0.76 [0.73, 0.78] 67% 0.9 69% 71% 95% 23% 7.8% 3.6% 63% 26% Protein Total 27 0.76 [0.73, 0.78] 67% 0.8 63% 75% 87% 44% 15% 8.7% 58% 19% Folic Acid 26 0.63 [0.60, 0.66] 31% 0.95 80% 34% 88% 23% 16% 3.9% 28% 53% Folic Acid 26 0.63 [0.60, 0.66] 31% 0.9 78% 39% 71% 48% 33% 9.2% 22% 35% Folic Acid 26 0.63 [0.60, 0.66] 31% 0.8 73% 47% 31% 84% 58% 22% 9.6% 11% Folic Acid 26 0.63 [0.60, 0.66] 31% 0.99 77% 32% 99% 1.0% 0.70% 0.21% 31% 68% Glucose 26 0.74 [0.69, 0.79] 9.5% 0.8 91% - 0 100% 91% 9.5% 0 0 Glucose 26 0.74 [0.69, 0.79] 9.5% 0.9 91% - 0 100% 91% 9.5% 0 0 Glucose 26 0.74 [0.69, 0.79] 9.5% 0.95 93% 61% 28% 98% 89% 6.8% 2.6% 1.7% Glucose 26 0.74 [0.69, 0.79] 9.5% 0.99 94% 22% 55% 79% 72% 4.3% 5.2% 19% Respiratory Culture 24 0.67 [0.64, 0.71] 33% 0.99 82% 33% 98% 5.5% 3.7% 0.81% 32% 64% Respiratory Culture 24 0.67 [0.64, 0.70] 33% 0.95 80% 39% 80% 38% 26% 6.4% 26% 42% Respiratory Culture 24 0.67 [0.64, 0.70] 33% 0.9 78% 44% 64% 62% 42% 12% 21% 26% Respiratory Culture 24 0.67 [0.64, 0.70] 33% 0.8 73% 60% 31% 90% 61% 22% 10% 6.6% CSF Culture And Gram 23 0.6 [0.48, 0.72] 3.7% 0.8 96% - 0 100% 96% 3.7% 0 0 Stain CSF Culture And Gram 23 0.6 [0.49, 0.72] 3.7% 0.95 96% - 0 100% 96% 3.7% 0 0 Stain CSF Culture And Gram 23 0.6 [0.48, 0.72] 3.7% 0.99 97% 10% 10% 97% 93% 3.3% 0.37% 3.3% Stain CSF Culture And Gram 23 0.6 [0.50, 0.71] 3.7% 0.9 96% - 0 100% 96% 3.7% 0 0 Stain Occult Bld 23 0.76 [0.71, 0.81] 76% 0.8 70% 79% 97% 19% 4.5% 1.9% 74% 19% Occult Bld 23 0.76 [0.71, 0.81] 76% 0.95 - 76% 100% 0 0 0 76% 24% Occult Bld 23 0.76 [0.71, 0.81] 76% 0.99 - 76% 100% 0 0 0 76% 24% Occult Bld 23 0.76 [0.71, 0.81] 76% 0.9 78% 77% 100% 5.1% 1.2% 0.35% 76% 23% iSTAT Creatinine 11 0.72 [0.69, 0.75] 37% 0.8 74% 53% 59% 69% 43% 15% 22% 20% iSTAT Creatinine 11 0.72 [0.69, 0.74] 37% 0.9 82% 46% 85% 40% 25% 5.7% 31% 37% iSTAT Creatinine 11 0.72 [0.69, 0.75] 37% 0.95 87% 43% 94% 26% 16% 2.3% 35% 47% iSTAT Creatinine 11 0.72 [0.69, 0.74] 37% 0.99 93% 40% 98% 14% 8.8% 0.71% 36% 54% iSTAT Cg4 10 0.76 [0.69, 0.84] 13% 0.8 87% - 0 100% 87% 13% 0 0 iSTAT Cg4 10 0.76 [0.68, 0.84] 13% 0.9 88% 100% 11% 100% 87% 12% 1.5% 0 iSTAT Cg4 10 0.76 [0.69, 0.84] 13% 0.95 91% 49% 42% 93% 81% 7.7% 5.6% 5.9% iSTAT Cg4 10 0.76 [0.69, 0.84] 13% 0.99 93% 25% 67% 69% 60% 4.4% 8.9% 27% Anti-hiv 1 0.58 [0.41, 0.75] 1.2% 0.99 99% 2.8% 44% 81% 80% 0.69% 0.54% 19% Anti-hiv 1 0.58 [0.42, 0.74] 1.2% 0.95 99% - 0 100% 99% 1.2% 0 0 Anti-hiv 1 0.58 [0.43, 0.75] 1.2% 0.9 99% - 0 100% 99% 1.2% 0 0 Anti-hiv 1 0.58 [0.41, 0.74] 1.2% 0.8 99% - 0 100% 99% 1.2% 0 0 Stool Culture 0 0.68 [0.57, 0.79] 9.0% 0.99 100% 9.3% 100% 3.4% 3.1% 0 9.0% 88% Stool Culture 0 0.68 [0.57, 0.79] 9.0% 0.95 95% 11% 78% 38% 34% 2.0% 7.0% 57% Stool Culture 0 0.68 [0.57, 0.79] 9.0% 0.9 92% 50% 8.7% 99% 90% 8.2% 0.78% 0.78% © 2019 Xu S et al. JAMA Network Open. Stool Culture 0 0.68 [0.57, 0.80] 9.0% 0.8 91% - 0 100% 91% 9.0% 0 0 eTable 3. Diagnostic Metrics for Common Stanford Standalone Labs Lab Test Vol C 95% CI Prev Target NPV Sens Spec NPV Potassium 9546 0.76 [0.73, 0.79] 13% 0.99 98% 13% 100% 2.5% 2.2% 0.04% 13% 85% Potassium 9546 0.76 [0.73, 0.79] 13% 0.8 87% - 0 100% 87% 13% 0 0 Potassium 9546 0.76 [0.73, 0.79] 13% 0.9 89% 48% 19% 97% 84% 11% 2.4% 2.6% Potassium 9546 0.76 [0.73, 0.79] 13% 0.95 95% 25% 77% 66% 57% 3.0% 10% 30% Hemoglobin 8016 0.94 [0.92, 0.95] 88% 0.99 85% 89% 100% 7.9% 0.93% 0.17% 88% 11% Hemoglobin 8016 0.94 [0.92, 0.95] 88% 0.95 81% 90% 99% 17% 2.1% 0.50% 88% 9.8% Hemoglobin 8016 0.94 [0.92, 0.95] 88% 0.9 77% 92% 99% 36% 4.3% 1.3% 87% 7.6% Hemoglobin 8016 0.94 [0.92, 0.95] 88% 0.8 69% 95% 96% 59% 6.9% 3.2% 85% 4.9% Sodium 7289 0.87 [0.85, 0.88] 42% 0.99 98% 42% 100% 3.0% 1.7% 0.04% 42% 57% Sodium 7289 0.87 [0.85, 0.88] 42% 0.8 75% 87% 56% 94% 55% 18% 23% 3.5% Sodium 7289 0.87 [0.85, 0.88] 42% 0.9 86% 68% 84% 71% 41% 6.5% 35% 17% Sodium 7289 0.87 [0.85, 0.88] 42% 0.95 92% 53% 95% 41% 24% 2.1% 40% 34% Creatinine 7012 0.96 [0.96, 0.97] 40% 0.99 99% 63% 99% 61% 36% 0.49% 40% 23% Creatinine 7012 0.96 [0.96, 0.97] 40% 0.95 94% 79% 93% 83% 50% 2.9% 37% 10.0% Creatinine 7012 0.96 [0.96, 0.97] 40% 0.9 90% 89% 85% 93% 56% 6.0% 34% 4.1% Creatinine 7012 0.96 [0.96, 0.97] 40% 0.8 80% 98% 64% 99% 59% 15% 26% 0.56% Urea Nitrogen 6999 0.95 [0.94, 0.96] 30% 0.8 81% 96% 45% 99% 69% 16% 13% 0.63% Urea Nitrogen 6999 0.95 [0.94, 0.96] 30% 0.9 89% 90% 73% 97% 68% 8.0% 22% 2.4% Urea Nitrogen 6999 0.95 [0.94, 0.96] 30% 0.99 99% 42% 99% 41% 29% 0.27% 30% 41% Urea Nitrogen 6999 0.95 [0.93, 0.96] 30% 0.95 94% 77% 87% 89% 62% 4.0% 26% 7.9% Calcium 6963 0.89 [0.87, 0.90] 59% 0.99 100% 60% 100% 3.9% 1.6% 0 59% 39% Calcium 6963 0.89 [0.87, 0.90] 59% 0.95 92% 65% 99% 24% 9.8% 0.80% 58% 31% Calcium 6963 0.89 [0.87, 0.90] 59% 0.9 88% 71% 96% 45% 18% 2.6% 57% 23% Calcium 6963 0.89 [0.87, 0.90] 59% 0.8 79% 82% 87% 73% 30% 7.9% 51% 11% CO2 6929 0.86 [0.84, 0.88] 19% 0.99 95% 35% 88% 60% 48% 2.3% 17% 32% CO2 6929 0.86 [0.84, 0.88] 19% 0.95 93% 54% 72% 85% 69% 5.5% 14% 12% CO2 6929 0.86 [0.85, 0.88] 19% 0.9 88% 83% 47% 98% 79% 10% 9.0% 1.9% CO2 6929 0.86 [0.84, 0.88] 19% 0.8 81% - 0 100% 81% 19% 0 0 Platelet Count 6660 0.91 [0.90, 0.92] 45% 0.8 76% 88% 64% 93% 51% 16% 29% 3.8% Platelet Count 6660 0.91 [0.90, 0.92] 45% 0.9 89% 80% 88% 82% 45% 5.6% 40% 9.8% Platelet Count 6660 0.91 [0.90, 0.92] 45% 0.95 92% 72% 93% 70% 39% 3.1% 42% 16% Platelet Count 6660 0.91 [0.90, 0.92] 45% 0.99 98% 59% 99% 42% 23% 0.38% 45% 32% © 2019 Xu S et al. JAMA Network Open. White Blood 6422 0.89 [0.88, 0.91] 45% 0.99 100% 45% 100% 0.54% 0.30% 0 45% 55% Cells White Blood 6422 0.89 [0.88, 0.91] 45% 0.95 93% 60% 95% 48% 27% 2.1% 42% 29% Cells White Blood 6422 0.89 [0.88, 0.90] 45% 0.9 87% 70% 87% 70% 39% 5.9% 39% 16% Cells White Blood 6422 0.89 [0.88, 0.91] 45% 0.8 78% 88% 67% 93% 51% 15% 30% 4.0% Cells Albumin 3063 0.93 [0.91, 0.94] 84% 0.99 - 84% 100% 0 0 0 84% 16% Albumin 3063 0.93 [0.91, 0.94] 84% 0.8 - 84% 100% 0 0 0 84% 16% Albumin 3063 0.93 [0.91, 0.94] 84% 0.9 - 84% 100% 0 0 0 84% 16% Albumin 3063 0.93 [0.91, 0.94] 84% 0.95 - 84% 100% 0 0 0 84% 16% Total Bilirubin 3038 0.97 [0.96, 0.98] 29% 0.9 87% 99% 65% 100% 71% 10% 19% 0.14% Total Bilirubin 3038 0.97 [0.96, 0.97] 29% 0.8 74% 100% 14% 100% 71% 25% 4.0% 0 Total Bilirubin 3038 0.97 [0.96, 0.98] 29% 0.99 98% 58% 96% 72% 51% 1.1% 28% 20% Total Bilirubin 3038 0.97 [0.96, 0.98] 29% 0.95 94% 92% 84% 97% 69% 4.8% 24% 2.0% Protein 2993 0.91 [0.89, 0.92] 31% 0.8 83% 90% 57% 97% 67% 13% 18% 2.0% Protein 2993 0.91 [0.89, 0.92] 31% 0.9 90% 74% 78% 88% 60% 6.8% 24% 8.6% Protein 2993 0.91 [0.89, 0.92] 31% 0.95 93% 59% 89% 72% 50% 3.5% 28% 19% Protein 2993 0.91 [0.90, 0.92] 31% 0.99 97% 39% 98% 31% 21% 0.72% 31% 48% AST (SGOT) 2986 0.92 [0.91, 0.93] 35% 0.95 94% 61% 92% 69% 45% 2.8% 32% 21% AST (SGOT) 2986 0.92 [0.91, 0.93] 35% 0.99 97% 44% 98% 34% 22% 0.70% 34% 43% AST (SGOT) 2986 0.92 [0.91, 0.93] 35% 0.9 89% 78% 80% 88% 57% 6.8% 28% 8.0% AST (SGOT) 2986 0.92 [0.91, 0.93] 35% 0.8 81% 96% 57% 99% 65% 15% 20% 0.83% ALT (SGPT) 2986 0.93 [0.92, 0.94] 30% 0.99 98% 45% 98% 47% 33% 0.74% 30% 37% ALT (SGPT) 2986 0.93 [0.92, 0.94] 30% 0.95 93% 71% 85% 85% 59% 4.7% 26% 11% ALT (SGPT) 2986 0.93 [0.92, 0.94] 30% 0.9 89% 94% 71% 98% 68% 8.8% 22% 1.5% ALT (SGPT) 2986 0.93 [0.92, 0.94] 30% 0.8 76% 99% 30% 100% 69% 21% 9.0% 0.12% Alk Phos 2984 0.94 [0.93, 0.95] 45% 0.8 69% 99% 45% 100% 55% 25% 20% 0.22% Alk Phos 2984 0.94 [0.93, 0.95] 45% 0.95 90% 77% 89% 78% 43% 4.8% 40% 12% Alk Phos 2984 0.94 [0.93, 0.95] 45% 0.99 97% 60% 98% 47% 26% 0.84% 44% 29% Alk Phos 2984 0.94 [0.93, 0.95] 45% 0.9 86% 91% 81% 93% 52% 8.7% 36% 3.7% eTable 4. Diagnostic Metrics for Common Stanford Components Lab Test C 95% CI Prev Target NPV Sens Spec NPV Magnesium 0.83 [0.80, 0.86] 9.1% 0.99 97% 16% 81% 59% 53% 1.7% 7.4% 38% Magnesium 0.83 [0.79, 0.86] 9.1% 0.95 94% 61% 41% 97% 89% 5.4% 3.7% 2.3% Magnesium 0.83 [0.80, 0.86] 9.1% 0.9 91% - 0 100% 91% 9.1% 0 0 Magnesium 0.83 [0.79, 0.86] 9.1% 0.8 91% - 0 100% 91% 9.1% 0 0 © 2019 Xu S et al. JAMA Network Open. Phosphorus 0.75 [0.73, 0.77] 31% 0.99 95% 33% 99% 6.5% 4.5% 0.23% 31% 64% Phosphorus 0.75 [0.73, 0.78] 31% 0.95 90% 36% 94% 25% 17% 2.0% 29% 51% Phosphorus 0.75 [0.73, 0.78] 31% 0.9 86% 43% 81% 52% 35% 5.9% 25% 33% Phosphorus 0.75 [0.73, 0.77] 31% 0.8 78% 62% 46% 87% 60% 17% 15% 8.9% Hemoglobin A1c 0.69 [0.64, 0.73] 71% 0.99 87% 72% 100% 6.9% 2.0% 0.31% 70% 27% Hemoglobin A1c 0.69 [0.64, 0.73] 71% 0.95 57% 74% 94% 18% 5.3% 4.0% 67% 24% Hemoglobin A1c 0.69 [0.64, 0.73] 71% 0.9 52% 74% 91% 24% 7.0% 6.3% 64% 22% Hemoglobin A1c 0.69 [0.64, 0.73] 71% 0.8 50% 77% 84% 39% 11% 11% 59% 18% Uric Acid 0.88 [0.84, 0.92] 38% 0.99 96% 45% 98% 25% 16% 0.64% 38% 46% Uric Acid 0.88 [0.84, 0.92] 38% 0.95 92% 63% 91% 67% 41% 3.5% 35% 20% Uric Acid 0.88 [0.84, 0.92] 38% 0.9 86% 72% 79% 81% 50% 8.0% 30% 12% Uric Acid 0.88 [0.84, 0.92] 38% 0.8 80% 88% 61% 95% 58% 15% 23% 3.2% Albumin 0.76 [0.65, 0.86] 69% 0.99 50% 70% 96% 8.3% 2.6% 2.6% 66% 29% Albumin 0.76 [0.65, 0.86] 69% 0.95 50% 72% 89% 25% 7.8% 7.8% 61% 23% Albumin 0.76 [0.65, 0.86] 69% 0.9 54% 79% 79% 54% 17% 14% 55% 14% Albumin 0.76 [0.66, 0.86] 69% 0.8 45% 84% 58% 75% 23% 29% 40% 7.8% Thyroid Stimulating 0.67 [0.63, 0.70] 15% 0.99 92% 17% 87% 27% 23% 1.9% 13% 62% Hormone Thyroid Stimulating 0.67 [0.63, 0.71] 15% 0.95 89% 24% 46% 75% 64% 8.0% 6.7% 21% Hormone Thyroid Stimulating 0.67 [0.62, 0.71] 15% 0.9 88% 47% 24% 95% 81% 11% 3.6% 4.1% Hormone Thyroid Stimulating 0.67 [0.63, 0.71] 15% 0.8 85% - 0 100% 85% 15% 0 0 Hormone Troponin I 0.95 [0.94, 0.96] 32% 0.99 96% 62% 94% 73% 50% 1.8% 30% 18% Troponin I 0.95 [0.94, 0.96] 32% 0.95 93% 80% 85% 90% 61% 4.9% 27% 7.0% Troponin I 0.95 [0.94, 0.96] 32% 0.9 90% 90% 77% 96% 65% 7.5% 25% 2.7% Troponin I 0.95 [0.94, 0.96] 32% 0.8 80% 100% 46% 100% 68% 17% 15% 0 Potassium 0.7 [0.63, 0.78] 38% 0.99 84% 45% 90% 33% 21% 3.9% 34% 42% Potassium 0.7 [0.62, 0.78] 38% 0.95 76% 52% 69% 61% 38% 12% 26% 24% Potassium 0.7 [0.62, 0.77] 38% 0.9 74% 60% 55% 77% 48% 17% 21% 14% Potassium 0.7 [0.62, 0.78] 38% 0.8 68% 66% 28% 91% 57% 27% 11% 5.6% Sodium 0.94 [0.90, 0.98] 51% 0.99 86% 94% 85% 95% 46% 7.6% 43% 2.5% Sodium 0.94 [0.90, 0.98] 51% 0.95 83% 98% 80% 99% 48% 10% 41% 0.64% Sodium 0.94 [0.90, 0.98] 51% 0.9 72% 98% 64% 99% 48% 18% 32% 0.64% Sodium 0.94 [0.90, 0.98] 51% 0.8 59% 97% 35% 99% 48% 33% 18% 0.64% Calcium 0.81 [0.69, 0.91] 63% 0.99 64% 95% 69% 93% 34% 20% 44% 2.4% Calcium 0.81 [0.71, 0.91] 63% 0.95 64% 95% 69% 93% 34% 20% 44% 2.4% Calcium 0.81 [0.69, 0.91] 63% 0.9 64% 95% 69% 93% 34% 20% 44% 2.4% Calcium 0.81 [0.71, 0.91] 63% 0.8 64% 95% 69% 93% 34% 20% 44% 2.4% © 2019 Xu S et al. JAMA Network Open. eTable 5. Diagnostic Metrics for Top UMich Standalone Labs Lab Test C 95% CI Prev Target NPV Sens Spec NPV White Blood 0.83 [0.81, 0.84] 46% 0.99 94% 51% 99% 19% 10% 0.66% 45% 44% Cells White Blood 0.83 [0.81, 0.84] 46% 0.95 89% 55% 95% 34% 18% 2.2% 44% 36% Cells White Blood 0.83 [0.81, 0.85] 46% 0.9 84% 61% 88% 53% 28% 5.6% 41% 25% Cells White Blood 0.83 [0.81, 0.85] 46% 0.8 75% 74% 70% 78% 42% 14% 32% 12% Cells Hemoglobin 0.94 [0.93, 0.95] 77% 0.99 85% 79% 99% 14% 3.2% 0.58% 76% 20% Hemoglobin 0.94 [0.93, 0.95] 77% 0.95 82% 81% 99% 22% 5.0% 1.1% 76% 18% Hemoglobin 0.94 [0.93, 0.95] 77% 0.9 80% 83% 98% 33% 7.7% 1.9% 75% 15% Hemoglobin 0.94 [0.93, 0.95] 77% 0.8 73% 89% 93% 62% 14% 5.2% 72% 8.9% Platelet Count 0.92 [0.90, 0.93] 32% 0.99 98% 41% 98% 34% 23% 0.48% 31% 45% Platelet Count 0.92 [0.91, 0.93] 32% 0.95 93% 66% 88% 79% 54% 3.9% 28% 14% Platelet Count 0.92 [0.90, 0.93] 32% 0.9 89% 83% 74% 93% 64% 8.1% 23% 4.6% Platelet Count 0.92 [0.90, 0.93] 32% 0.8 79% 98% 41% 100% 68% 18% 13% 0.29% Sodium 0.92 [0.90, 0.93] 22% 0.99 - 22% 100% 0 0 0 22% 78% Sodium 0.92 [0.90, 0.93] 22% 0.95 96% 50% 88% 76% 60% 2.5% 19% 19% Sodium 0.92 [0.90, 0.93] 22% 0.9 93% 84% 72% 96% 76% 6.1% 15% 2.9% Sodium 0.92 [0.90, 0.93] 22% 0.8 78% - 0 100% 78% 22% 0 0 Potassium 0.76 [0.74, 0.79] 13% 0.99 100% 13% 100% 2.1% 1.8% 0 13% 85% Potassium 0.76 [0.74, 0.79] 13% 0.95 95% 21% 83% 54% 47% 2.3% 11% 40% Potassium 0.76 [0.74, 0.79] 13% 0.9 91% 40% 36% 92% 80% 8.3% 4.8% 7.0% Potassium 0.76 [0.74, 0.79] 13% 0.8 87% - 0 100% 87% 13% 0 0 Creatinine 0.9 [0.89, 0.91] 41% 0.99 95% 50% 97% 32% 19% 1.0% 40% 40% Creatinine 0.9 [0.89, 0.91] 41% 0.95 91% 63% 91% 62% 36% 3.7% 38% 22% Creatinine 0.9 [0.88, 0.91] 41% 0.9 87% 77% 82% 83% 48% 7.4% 34% 10% Creatinine 0.9 [0.89, 0.91] 41% 0.8 77% 92% 60% 96% 56% 16% 25% 2.2% Total Bilirubin 0.93 [0.91, 0.94] 28% 0.99 94% 63% 86% 81% 58% 3.9% 24% 14% Total Bilirubin 0.93 [0.91, 0.94] 28% 0.95 92% 87% 79% 96% 69% 5.8% 22% 3.1% Total Bilirubin 0.93 [0.91, 0.94] 28% 0.9 88% 99% 65% 100% 72% 9.7% 18% 0.23% Total Bilirubin 0.93 [0.91, 0.94] 28% 0.8 77% 100% 22% 100% 72% 22% 6.0% 0 CO2 0.87 [0.84, 0.89] 15% 0.99 100% 15% 100% 0.37% 0.31% 0 15% 85% CO2 0.87 [0.84, 0.89] 15% 0.95 94% 52% 68% 89% 76% 4.9% 10% 9.3% CO2 0.87 [0.85, 0.89] 15% 0.9 88% 83% 24% 99% 84% 11% 3.7% 0.74% © 2019 Xu S et al. JAMA Network Open. CO2 0.87 [0.85, 0.89] 15% 0.8 85% - 0 100% 85% 15% 0 0 AST (SGOT) 0.88 [0.86, 0.89] 48% 0.99 95% 50% 100% 6.8% 3.5% 0.20% 48% 48% AST (SGOT) 0.88 [0.87, 0.89] 48% 0.95 93% 56% 98% 29% 15% 1.1% 47% 37% AST (SGOT) 0.88 [0.87, 0.89] 48% 0.9 90% 61% 94% 45% 23% 2.6% 45% 28% AST (SGOT) 0.88 [0.87, 0.89] 48% 0.8 79% 79% 77% 81% 42% 11% 37% 9.8% ALT (SGPT) 0.92 [0.91, 0.93] 40% 0.99 99% 45% 100% 21% 13% 0.19% 40% 48% ALT (SGPT) 0.92 [0.91, 0.93] 40% 0.95 96% 59% 96% 56% 34% 1.5% 38% 26% ALT (SGPT) 0.92 [0.91, 0.93] 40% 0.9 90% 68% 88% 73% 44% 4.7% 35% 17% ALT (SGPT) 0.92 [0.91, 0.93] 40% 0.8 82% 95% 68% 97% 59% 13% 27% 1.6% Albumin 0.9 [0.89, 0.92] 43% 0.99 97% 49% 99% 20% 12% 0.32% 43% 45% Albumin 0.9 [0.89, 0.92] 43% 0.95 93% 56% 96% 43% 24% 1.9% 42% 32% Albumin 0.9 [0.89, 0.92] 43% 0.9 87% 73% 86% 75% 43% 6.3% 37% 14% Albumin 0.9 [0.89, 0.92] 43% 0.8 77% 93% 62% 96% 55% 16% 27% 2.0% Calcium 0.89 [0.88, 0.90] 35% 0.99 98% 40% 99% 20% 13% 0.20% 35% 52% Calcium 0.89 [0.88, 0.90] 35% 0.95 93% 55% 92% 59% 39% 2.8% 32% 27% Calcium 0.89 [0.88, 0.90] 35% 0.9 89% 68% 81% 80% 52% 6.7% 28% 13% Calcium 0.89 [0.88, 0.90] 35% 0.8 81% 88% 57% 96% 62% 15% 20% 2.8% Protein 0.91 [0.90, 0.92] 43% 0.99 99% 46% 100% 11% 6.0% 0.07% 43% 51% Protein 0.91 [0.90, 0.92] 43% 0.95 93% 55% 96% 39% 22% 1.6% 42% 34% Protein 0.91 [0.90, 0.92] 43% 0.9 89% 76% 87% 80% 45% 5.5% 38% 12% Protein 0.91 [0.90, 0.92] 43% 0.8 79% 94% 66% 97% 55% 15% 28% 1.9% Alk Phos 0.92 [0.91, 0.93] 27% 0.99 99% 39% 99% 43% 31% 0.27% 27% 42% Alk Phos 0.92 [0.91, 0.93] 27% 0.95 94% 58% 87% 77% 56% 3.5% 24% 17% Alk Phos 0.92 [0.91, 0.93] 27% 0.9 90% 83% 71% 94% 69% 7.8% 20% 4.0% Alk Phos 0.92 [0.91, 0.93] 27% 0.8 83% 99% 48% 100% 73% 14% 13% 0.12% Urea Nitrogen 0.93 [0.92, 0.94] 48% 0.99 97% 55% 99% 26% 13% 0.41% 48% 39% Urea Nitrogen 0.93 [0.92, 0.94] 48% 0.95 93% 69% 95% 60% 31% 2.2% 46% 21% Urea Nitrogen 0.93 [0.92, 0.94] 48% 0.9 89% 81% 90% 80% 42% 5.0% 43% 10% Urea Nitrogen 0.93 [0.92, 0.94] 48% 0.8 78% 94% 71% 96% 50% 14% 34% 2.3% eTable 6. Diagnostic Metrics for Common UMich Components Lab Test C 95% CI Prev Target NPV Sens Spec NPV Magnesium 0.81 [0.79, 0.83] 21% 0.99 - 21% 100% 0 0 0 21% 79% Magnesium 0.81 [0.78, 0.83] 21% 0.95 95% 32% 90% 48% 37% 2.1% 19% 41% Magnesium 0.81 [0.78, 0.83] 21% 0.9 90% 42% 71% 73% 58% 6.1% 15% 21% Magnesium 0.81 [0.79, 0.83] 21% 0.8 82% 79% 21% 98% 77% 17% 4.5% 1.2% Phosphorus 0.8 [0.78, 0.82] 24% 0.99 97% 26% 99% 9.4% 7.1% 0.24% 24% 69% Phosphorus 0.8 [0.78, 0.82] 24% 0.95 91% 39% 81% 60% 46% 4.5% 20% 30% © 2019 Xu S et al. JAMA Network Open. Phosphorus 0.8 [0.78, 0.82] 24% 0.9 88% 55% 62% 84% 64% 9.0% 15% 12% Phosphorus 0.8 [0.78, 0.82] 24% 0.8 79% 88% 15% 99% 75% 20% 3.7% 0.52% Prothrombin Time 0.93 [0.92, 0.94] 44% 0.99 97% 50% 99% 24% 13% 0.40% 44% 43% Prothrombin Time 0.93 [0.92, 0.94] 44% 0.95 92% 69% 93% 67% 38% 3.2% 41% 18% Prothrombin Time 0.93 [0.92, 0.94] 44% 0.9 87% 83% 84% 87% 49% 7.1% 37% 7.5% Prothrombin Time 0.93 [0.92, 0.94] 44% 0.8 78% 96% 66% 98% 55% 15% 29% 1.1% Partial Thromboplastin Time 0.92 [0.91, 0.93] 49% 0.99 93% 60% 97% 38% 20% 1.6% 48% 31% Partial Thromboplastin Time 0.92 [0.91, 0.93] 49% 0.95 89% 76% 91% 72% 36% 4.5% 45% 14% Partial Thromboplastin Time 0.92 [0.91, 0.93] 49% 0.9 86% 84% 86% 84% 43% 7.0% 42% 8.1% Partial Thromboplastin Time 0.92 [0.91, 0.93] 49% 0.8 81% 94% 77% 95% 48% 11% 38% 2.4% Alkaline Phosphatase 0.9 [0.89, 0.91] 46% 0.99 96% 55% 99% 30% 16% 0.64% 46% 37% Alkaline Phosphatase 0.9 [0.89, 0.91] 46% 0.95 93% 63% 95% 51% 27% 2.2% 44% 26% Alkaline Phosphatase 0.9 [0.89, 0.91] 46% 0.9 85% 72% 85% 71% 38% 6.8% 40% 16% Alkaline Phosphatase 0.9 [0.89, 0.91] 46% 0.8 78% 90% 70% 93% 50% 14% 32% 3.8% Sodium 0.87 [0.86, 0.89] 41% 0.99 - 41% 100% 0 0 0 41% 59% Sodium 0.87 [0.86, 0.89] 41% 0.95 92% 55% 94% 45% 26% 2.3% 39% 32% Sodium 0.87 [0.86, 0.89] 41% 0.9 87% 64% 86% 66% 39% 5.7% 36% 20% Sodium 0.87 [0.86, 0.89] 41% 0.8 78% 86% 62% 93% 54% 16% 26% 4.3% Potassium 0.76 [0.74, 0.79] 17% 0.99 96% 22% 94% 31% 26% 1.1% 16% 57% Potassium 0.76 [0.73, 0.79] 17% 0.95 92% 30% 71% 66% 55% 4.9% 12% 28% Potassium 0.76 [0.73, 0.79] 17% 0.9 88% 44% 37% 91% 75% 11% 6.2% 7.8% Potassium 0.76 [0.74, 0.79] 17% 0.8 83% - 0 100% 83% 17% 0 0 Troponin I 0.89 [0.88, 0.91] 44% 0.99 96% 48% 99% 15% 8.6% 0.36% 44% 47% Troponin I 0.89 [0.88, 0.91] 44% 0.95 91% 61% 94% 52% 29% 2.8% 42% 27% Troponin I 0.89 [0.88, 0.91] 44% 0.9 86% 71% 86% 72% 40% 6.3% 38% 16% Troponin I 0.89 [0.88, 0.91] 44% 0.8 80% 86% 72% 91% 50% 12% 32% 5.2% Lactate Dehydrogenase 0.94 [0.93, 0.95] 66% 0.99 97% 72% 100% 25% 8.2% 0.27% 66% 25% Lactate Dehydrogenase 0.94 [0.93, 0.95] 66% 0.95 94% 78% 99% 45% 15% 0.97% 65% 18% Lactate Dehydrogenase 0.94 [0.93, 0.95] 66% 0.9 87% 82% 96% 58% 20% 2.8% 64% 14% Lactate Dehydrogenase 0.94 [0.93, 0.95] 66% 0.8 76% 90% 87% 81% 27% 8.5% 58% 6.2% Calcium, Ionized 0.86 [0.84, 0.87] 63% 0.99 92% 64% 100% 3.7% 1.4% 0.11% 63% 36% Calcium, Ionized 0.86 [0.84, 0.87] 63% 0.95 89% 69% 98% 24% 9.0% 1.2% 62% 28% Calcium, Ionized 0.86 [0.84, 0.87] 63% 0.9 83% 75% 95% 45% 17% 3.4% 60% 20% Calcium, Ionized 0.86 [0.85, 0.87] 63% 0.8 74% 85% 84% 75% 28% 9.9% 53% 9.2% Uric Acid 0.91 [0.90, 0.93] 46% 0.99 97% 54% 99% 30% 17% 0.49% 45% 38% Uric Acid 0.91 [0.90, 0.93] 46% 0.95 94% 61% 97% 48% 26% 1.5% 44% 28% Uric Acid 0.91 [0.90, 0.93] 46% 0.9 90% 71% 90% 70% 38% 4.4% 41% 16% Uric Acid 0.91 [0.89, 0.93] 46% 0.8 83% 84% 79% 88% 48% 9.7% 36% 6.7% Albumin 0.85 [0.82, 0.87] 79% 0.99 100% 80% 100% 3.5% 0.73% 0 79% 20% © 2019 Xu S et al. JAMA Network Open. Albumin 0.85 [0.82, 0.87] 79% 0.95 54% 80% 99% 6.6% 1.4% 1.2% 78% 19% Albumin 0.85 [0.82, 0.87] 79% 0.9 58% 82% 97% 18% 3.8% 2.7% 77% 17% Albumin 0.85 [0.82, 0.87] 79% 0.8 56% 85% 92% 40% 8.3% 6.4% 73% 12% Thyroid Stimulating Hormone 0.66 [0.61, 0.71] 33% 0.99 80% 36% 88% 23% 15% 3.9% 29% 52% Thyroid Stimulating Hormone 0.66 [0.61, 0.71] 33% 0.95 76% 42% 65% 56% 37% 12% 21% 30% Thyroid Stimulating Hormone 0.66 [0.61, 0.71] 33% 0.9 74% 50% 42% 79% 53% 19% 14% 14% Thyroid Stimulating Hormone 0.66 [0.61, 0.71] 33% 0.8 71% 65% 19% 95% 64% 26% 6.4% 3.5% eTable 7. Diagnostic Metrics for Top UCSF Standalone Labs Lab Test C 95% CI Prev Target NPV Sens Spec NPV White Blood 0.89 [0.87, 0.90] 52% 0.99 94% 54% 99% 8.4% 4.0% 0.28% 51% 44% Cells White Blood 0.89 [0.87, 0.90] 52% 0.95 88% 63% 95% 41% 20% 2.6% 49% 28% Cells White Blood 0.89 [0.87, 0.90] 52% 0.9 85% 71% 90% 60% 29% 5.0% 47% 19% Cells White Blood 0.89 [0.87, 0.90] 52% 0.8 76% 86% 74% 88% 42% 14% 38% 6.0% Cells Hemoglobin 0.95 [0.94, 0.96] 89% 0.99 81% 90% 100% 4.3% 0.46% 0.11% 89% 10% Hemoglobin 0.95 [0.93, 0.96] 89% 0.95 76% 90% 100% 11% 1.1% 0.35% 89% 9.4% Hemoglobin 0.95 [0.94, 0.96] 89% 0.9 82% 92% 99% 25% 2.7% 0.60% 89% 7.8% Hemoglobin 0.95 [0.94, 0.96] 89% 0.8 73% 93% 98% 40% 4.3% 1.6% 88% 6.3% Platelet Count 0.95 [0.94, 0.96] 39% 0.99 97% 60% 98% 57% 35% 0.98% 38% 26% Platelet Count 0.95 [0.94, 0.96] 39% 0.95 92% 80% 89% 85% 52% 4.4% 35% 8.9% Platelet Count 0.95 [0.94, 0.96] 39% 0.9 88% 92% 79% 96% 58% 8.1% 31% 2.6% Platelet Count 0.95 [0.94, 0.96] 39% 0.8 76% 99% 50% 100% 60% 20% 20% 0.20% Sodium 0.89 [0.87, 0.90] 33% 0.99 100% 33% 100% 0.55% 0.37% 0 33% 67% Sodium 0.89 [0.88, 0.90] 33% 0.95 95% 49% 95% 53% 35% 1.7% 31% 32% Sodium 0.89 [0.88, 0.90] 33% 0.9 90% 64% 81% 78% 53% 6.1% 26% 15% Sodium 0.89 [0.87, 0.90] 33% 0.8 82% 86% 56% 96% 64% 14% 18% 3.0% Potassium 0.77 [0.75, 0.79] 16% 0.99 97% 20% 95% 28% 23% 0.84% 15% 61% Potassium 0.77 [0.74, 0.79] 16% 0.95 94% 28% 79% 61% 51% 3.4% 13% 33% Potassium 0.77 [0.74, 0.79] 16% 0.9 90% 41% 49% 87% 73% 8.3% 7.8% 11% Potassium 0.77 [0.74, 0.79] 16% 0.8 84% - 0 100% 84% 16% 0 0 Creatinine 0.94 [0.93, 0.95] 41% 0.99 96% 47% 99% 23% 14% 0.51% 40% 45% Creatinine 0.94 [0.93, 0.95] 41% 0.95 93% 72% 92% 76% 45% 3.2% 37% 14% Creatinine 0.94 [0.93, 0.95] 41% 0.9 89% 91% 83% 94% 56% 6.9% 34% 3.4% Creatinine 0.94 [0.93, 0.95] 41% 0.8 75% 98% 52% 99% 59% 19% 21% 0.47% Total Bilirubin 0.93 [0.91, 0.94] 30% 0.99 98% 44% 98% 48% 34% 0.70% 29% 37% © 2019 Xu S et al. JAMA Network Open. Total Bilirubin 0.93 [0.91, 0.94] 30% 0.95 93% 64% 87% 80% 56% 3.9% 26% 14% Total Bilirubin 0.93 [0.91, 0.94] 30% 0.9 89% 85% 72% 95% 67% 8.4% 21% 3.7% Total Bilirubin 0.93 [0.91, 0.94] 30% 0.8 81% 99% 45% 100% 70% 16% 13% 0.12% CO2 0.87 [0.86, 0.89] 21% 0.99 97% 24% 98% 13% 11% 0.37% 21% 68% CO2 0.87 [0.86, 0.89] 21% 0.95 95% 43% 86% 69% 55% 3.0% 18% 24% CO2 0.87 [0.86, 0.89] 21% 0.9 90% 67% 62% 92% 72% 8.1% 13% 6.6% CO2 0.87 [0.86, 0.89] 21% 0.8 79% 91% 1.7% 100% 79% 21% 0.37% 0.04% AST (SGOT) 0.85 [0.83, 0.86] 46% 0.99 94% 47% 100% 4.8% 2.6% 0.17% 46% 52% AST (SGOT) 0.85 [0.83, 0.86] 46% 0.95 90% 54% 96% 30% 16% 1.9% 44% 38% AST (SGOT) 0.85 [0.83, 0.86] 46% 0.9 86% 61% 90% 51% 28% 4.5% 41% 27% AST (SGOT) 0.85 [0.83, 0.86] 46% 0.8 77% 73% 72% 78% 42% 13% 33% 12% ALT (SGPT) 0.91 [0.90, 0.93] 31% 0.99 99% 41% 99% 36% 25% 0.35% 31% 44% ALT (SGPT) 0.91 [0.90, 0.93] 31% 0.95 93% 56% 89% 67% 46% 3.4% 28% 22% ALT (SGPT) 0.91 [0.90, 0.93] 31% 0.9 89% 80% 76% 91% 63% 7.7% 24% 6.0% ALT (SGPT) 0.91 [0.90, 0.93] 31% 0.8 78% 99% 39% 100% 69% 19% 12% 0.12% Albumin 0.91 [0.90, 0.92] 78% 0.99 100% 78% 100% 0.18% 0.04% 0 78% 22% Albumin 0.91 [0.90, 0.92] 78% 0.95 100% 78% 100% 0.18% 0.04% 0 78% 22% Albumin 0.91 [0.90, 0.92] 78% 0.9 79% 80% 99% 9.1% 2.0% 0.52% 78% 20% Albumin 0.91 [0.90, 0.92] 78% 0.8 75% 83% 97% 31% 6.8% 2.2% 76% 15% Calcium 0.88 [0.87, 0.89] 69% 0.99 88% 71% 99% 10% 3.2% 0.42% 69% 28% Calcium 0.88 [0.87, 0.90] 69% 0.95 88% 74% 99% 23% 7.2% 0.96% 68% 24% Calcium 0.88 [0.87, 0.90] 69% 0.9 85% 78% 97% 39% 12% 2.1% 67% 19% Calcium 0.88 [0.87, 0.90] 69% 0.8 75% 83% 91% 59% 18% 6.1% 63% 13% Protein 0.9 [0.89, 0.91] 54% 0.99 97% 57% 100% 12% 5.5% 0.16% 54% 40% Protein 0.9 [0.89, 0.91] 54% 0.95 93% 62% 98% 30% 14% 1.1% 53% 32% Protein 0.9 [0.89, 0.91] 54% 0.9 88% 69% 94% 51% 24% 3.2% 51% 22% Protein 0.9 [0.89, 0.91] 54% 0.8 81% 84% 83% 81% 37% 9.0% 45% 8.7% Alk Phos 0.93 [0.92, 0.94] 45% 0.99 98% 61% 99% 48% 26% 0.59% 45% 29% Alk Phos 0.93 [0.92, 0.94] 45% 0.95 92% 71% 93% 68% 37% 3.0% 42% 18% Alk Phos 0.93 [0.92, 0.94] 45% 0.9 87% 78% 86% 79% 43% 6.3% 39% 11% Alk Phos 0.93 [0.92, 0.94] 45% 0.8 79% 96% 69% 97% 53% 14% 31% 1.4% Urea Nitrogen 0.93 [0.91, 0.94] 38% 0.99 98% 47% 99% 31% 19% 0.38% 38% 43% Urea Nitrogen 0.93 [0.91, 0.94] 38% 0.95 92% 67% 90% 73% 45% 3.7% 34% 17% Urea Nitrogen 0.93 [0.91, 0.93] 38% 0.9 89% 81% 83% 88% 54% 6.6% 31% 7.5% Urea Nitrogen 0.93 [0.91, 0.94] 38% 0.8 79% 95% 58% 98% 61% 16% 22% 1.2% eTable 8. Diagnostic Metrics for Common UCSF Components Lab Test Stanford - Stanford - Stanford - UCSF - UCSF - UCSF - UMich - UMich - UMich - Stanford UCSF UMich Stanford UCSF UMich Stanford UCSF UMich © 2019 Xu S et al. JAMA Network Open. White Blood 0.89 0.88 0.79 0.87 0.88 0.81 0.87 0.88 0.83 Cells [0.88, 0.91] [0.86, 0.89] [0.77, 0.81] [0.86, 0.89] [0.87, 0.9] [0.8, 0.83] [0.86, 0.89] [0.86, 0.89] [0.81, 0.84] Hemoglobin 0.93 0.94 0.89 0.86 0.9 0.79 0.92 0.94 0.9 [0.92, 0.94] [0.93, 0.95] [0.88, 0.91] [0.84, 0.88] [0.89, 0.92] [0.77, 0.81] [0.9, 0.93] [0.93, 0.95] [0.89, 0.91] Platelet Count 0.91 0.94 0.91 0.89 0.95 0.91 0.89 0.94 0.92 [0.9, 0.92] [0.93, 0.95] [0.9, 0.93] [0.88, 0.91] [0.94, 0.96] [0.89, 0.92] [0.88, 0.91] [0.93, 0.95] [0.9, 0.93] Sodium 0.87 0.88 0.91 0.85 0.89 0.91 0.86 0.86 0.91 [0.85, 0.88] [0.86, 0.89] [0.9, 0.93] [0.84, 0.87] [0.87, 0.9] [0.89, 0.92] [0.84, 0.88] [0.84, 0.88] [0.9, 0.93] Potassium 0.76 0.75 0.67 0.74 0.77 0.75 0.73 0.75 0.76 [0.73, 0.79] [0.73, 0.78] [0.63, 0.7] [0.71, 0.77] [0.74, 0.79] [0.72, 0.78] [0.69, 0.76] [0.72, 0.77] [0.73, 0.79] CO2 0.86 0.8 0.75 0.8 0.87 0.85 0.77 0.82 0.87 [0.84, 0.88] [0.78, 0.82] [0.71, 0.78] [0.77, 0.82] [0.86, 0.89] [0.82, 0.87] [0.74, 0.8] [0.8, 0.84] [0.84, 0.88] Urea Nitrogen 0.95 0.92 0.9 0.94 0.92 0.9 0.93 0.92 0.92 [0.94, 0.96] [0.91, 0.93] [0.89, 0.92] [0.93, 0.95] [0.91, 0.93] [0.89, 0.92] [0.92, 0.94] [0.91, 0.93] [0.91, 0.93] Creatinine 0.96 0.91 0.85 0.94 0.94 0.88 0.92 0.88 0.9 [0.96, 0.97] [0.89, 0.92] [0.83, 0.86] [0.94, 0.95] [0.93, 0.95] [0.87, 0.9] [0.91, 0.93] [0.86, 0.89] [0.88, 0.91] Calcium 0.88 0.86 0.81 0.87 0.87 0.85 0.85 0.86 0.89 [0.87, 0.9] [0.85, 0.88] [0.79, 0.83] [0.85, 0.88] [0.86, 0.88] [0.83, 0.86] [0.83, 0.87] [0.84, 0.87] [0.88, 0.9] Albumin 0.92 0.88 0.73 0.84 0.89 0.74 0.92 0.89 0.9 [0.91, 0.93] [0.87, 0.9] [0.7, 0.75] [0.82, 0.86] [0.88, 0.9] [0.72, 0.76] [0.9, 0.93] [0.87, 0.9] [0.89, 0.92] Protein 0.91 0.89 0.87 0.88 0.89 0.85 0.89 0.88 0.9 [0.89, 0.92] [0.88, 0.9] [0.86, 0.88] [0.87, 0.9] [0.88, 0.9] [0.83, 0.86] [0.87, 0.9] [0.86, 0.89] [0.89, 0.91] Alk Phos 0.94 0.91 0.89 0.92 0.93 0.89 0.92 0.93 0.92 [0.93, 0.95] [0.9, 0.92] [0.88, 0.91] [0.91, 0.93] [0.92, 0.94] [0.87, 0.9] [0.91, 0.93] [0.92, 0.94] [0.91, 0.93] Total Bilirubin 0.96 0.91 0.91 0.95 0.93 0.91 0.96 0.92 0.92 [0.95, 0.97] [0.89, 0.92] [0.89, 0.93] [0.94, 0.97] [0.91, 0.94] [0.89, 0.92] [0.94, 0.97] [0.9, 0.93] [0.9, 0.93] AST (SGOT) 0.92 0.81 0.86 0.85 0.77 0.73 0.88 0.77 0.86 [0.91, 0.93] [0.8, 0.83] [0.85, 0.87] [0.83, 0.86] [0.76, 0.79] [0.71, 0.75] [0.87, 0.9] [0.75, 0.79] [0.85, 0.87] ALT (SGPT) 0.93 0.86 0.91 0.92 0.91 0.88 0.88 0.84 0.88 [0.92, 0.94] [0.84, 0.87] [0.9, 0.92] [0.91, 0.93] [0.9, 0.93] [0.86, 0.89] [0.87, 0.89] [0.82, 0.86] [0.87, 0.9] eTable 9. Diagnostic Metrics for Common Components in Transferability Study Lab Test Medicare Chargemaster Magnesium $8.27 $280.00 Prothrombin Time $4.85 $190.00 Phosphorus $5.85 $225.00 Partial Thromboplastin Time $7.98 $240.00 Lactate $11.87 $330.00 Calcium Ionized $13.73 $399.00 Potassium $5.68 $204.00 Troponin I $12.47 $520.00 LDH Total $6.71 $193.00 Heparin $16.16 $466.00 Urinalysis - $196.00 © 2019 Xu S et al. JAMA Network Open. Blood Culture (Aerobic & Anaerobic) - $499.00 Blood Culture (2 Aerobic) - $499.00 Sodium $5.94 $219.00 Lidocaine $18.14 $264.00 Hematocrit $2.93 $217.00 Urine Culture $9.96 $317.00 Urinalysis With Microscopic $3.76 $196.00 Uric Acid $5.58 $135.00 Hemoglobin A1c $11.99 $145.00 Sepsis Protocol Lactate $11.87 $330.00 iSTAT Troponin I - $520.00 Platelet Count $5.53 $138.00 Lipase $8.51 $213.00 Procalcitonin $33.08 $318.00 Lactic Acid $13.19 $330.00 Fibrinogen $14.02 $285.00 Thyroid Stimulating Hormone $20.75 $380.00 Creatine Kinase $8.04 $254.00 C-Reactive Protein $6.39 $179.00 -proBNP $41.90 $821.00 Triglycerides $7.09 $175.00 $14.26 $446.00 C.diff Toxin B Gene - $486.00 Osmolality $8.16 $209.00 Respiratory Culture And Gram Stain - $226.00 Sedimentation Rate (ESR) - $75.00 Ferritin $16.83 $375.00 Albumin - $138.00 Ammonia - $334.00 Specific Gravity $3.28 $115.00 Fungal Culture - $309.00 Haptoglobin - $258.00 Anaerobic Culture $11.66 $544.00 Fluid Culture And Gram Stain - $226.00 Cmv Dna Pcr Quant - $849.00 Blood Cult Central Line Catheter By Nurse - $499.00 - $246.00 Transferrin Saturation $15.76 $292.50 Vitamin B12 $18.61 $281.00 © 2019 Xu S et al. JAMA Network Open. Blood Cult - First Set - $499.00 Prealbumin $18.01 $246.00 Osmolality $8.42 $255.00 Digoxin - $389.00 Reticulocyte Count Automated $4.93 $179.00 Cortisol - $374.00 Hepatitis B Antigen $12.75 $220.00 Iron - $165.00 AFB Culture - $162.00 Calcium - $143.00 Biopsy/tissue With Gram Stain - $182.00 Afb Culture - $249.50 Pregnancy Test $8.61 $102.00 Urea Nitrogen - $157.00 Gram Stain - $150.00 - $747.00 Protein Total $21.24 $229.00 Folic Acid - $274.00 Glucose - $250.00 Respiratory Culture - $302.00 CSF Culture And Gram Stain - $226.00 Occult Bld - $130.00 iSTAT Creatinine $6.33 $179.00 iSTAT Cg4 - $572.50 Anti-hiv - $280.00 Stool Culture $11.66 $402.50 eTable 10. Medicare and Chargemaster Fees for Standalone Labs © 2019 Xu S et al. JAMA Network Open. lab feature 1 score 1 feature 2 score 2 feature 3 score 3 Hemoglobin A1c last_normality 0.437 Diabetes 0.154 HCT 0.054 AFB Culture last_normality 0.639 LABAFBC 0.132 Birth 0.087 Afb Culture AdmitDxDate 0.467 BP_Low_Diastolic 0.267 Pulse 0.2 Albumin ALB 0.434 last_normality 0.094 Temp 0.086 Anaerobic Culture last_normality 0.448 Temp 0.102 Pulse 0.078 Vitamin B12 ALB 0.136 TBIL 0.119 PLT 0.093 Blood Culture (Aerobic & Temp 0.178 Pulse 0.124 BP_High_Systolic 0.11 Anaerobic) Blood Culture (2 Aerobic) PLT 0.149 Pulse 0.119 last_normality 0.113 Blood Cult - First Set WBC 0.252 Temp 0.168 K 0.128 Blood Cult Central Line Catheter Pulse 0.211 Temp 0.168 TBIL 0.1 By Nurse Urea Nitrogen BUN 0.408 LABBUN 0.14 CR 0.067 Biopsy/tissue With Gram Stain last_normality 0.312 WBC 0.166 PLT 0.091 Calcium CA 0.634 Pulse 0.083 last_normality 0.08 Calcium Ionized CAION 0.344 last_normality 0.182 PHCAI 0.154 C.difficile Toxin B Gene LABCDTPCR 0.295 Pulse 0.137 last_normality 0.089 Creatine Kinase CK 0.411 last_normality 0.264 PHA 0.055 Cmv Dna Pcr Quant CMVLOG 0.266 last_normality 0.212 CMVCP 0.084 Cortisol K 0.187 LAC 0.121 CO2 0.117 C-Reactive Protein CRP 0.217 Temp 0.161 last_normality 0.152 CSF Culture And Gram Stain WBC 0.173 Pulse 0.14 Temp 0.139 Glucose GLUCSF 0.393 last_normality 0.109 NA 0.068 Protein Total TPCSF 0.191 Pulse 0.113 BP_High_Systolic 0.091 Digoxin last_normality 0.759 DIG 0.188 Pulse 0.024 Sedimentation Rate (ESR) HCT 0.269 last_normality 0.134 ALB 0.12 Fungal Culture last_normality 0.152 Resp 0.116 Pulse 0.111 Iron Total Temp 0.158 Male 0.125 PLT 0.121 Ferritin last_normality 0.345 ALB 0.175 Temp 0.072 Fibrinogen FIBRINOGEN 0.463 last_normality 0.237 PLT 0.046 Fluid Culture And Gram Stain Temp 0.175 SurgerySpecialty 0.134 Pulse 0.106 Folic Acid Birth 0.195 CA 0.129 BP_High_Systolic 0.103 © 2019 Xu S et al. JAMA Network Open. T4 Free BP_High_Systolic 0.155 last_normality 0.124 Temp 0.097 Gram Stain BP_High_Systolic 0.116 BP_Low_Diastolic 0.093 Resp 0.091 Haptoglobin last_normality 0.547 TBIL 0.08 PLT 0.058 Hepatitis B Antigen last_normality 0.302 AdmitDxDate 0.284 order_time 0.067 Hematocrit HCT 0.577 last_normality 0.114 Resp 0.061 Heparin last_normality 0.431 HEPAR 0.179 Resp 0.078 Anti-hiv PLT 0.255 HCT 0.196 Temp 0.101 Potassium K 0.678 last_normality 0.085 HCT 0.082 Lactic Acid LAC 0.504 last_normality 0.263 PHA 0.044 Lactate last_normality 0.707 LACWBL 0.138 Pulse 0.027 LDH Total last_normality 0.773 LDH 0.168 WBC 0.01 Lidocaine LIDO 0.329 last_normality 0.315 BP_Low_Diastolic 0.078 Lipase last_normality 0.66 Pulse 0.047 LIPASE 0.045 Ck Mb (mass) last_normality 0.31 CKMBRI 0.124 CKMB 0.109 Magnesium MG 0.471 last_normality 0.235 Resp 0.044 Sodium last_normality 0.659 NA 0.295 Pulse 0.009 Ammonia NH3 0.362 last_normality 0.202 PLT 0.076 Nt - Probnp last_normality 0.318 Birth 0.204 BUN 0.094 Osmolality NA 0.338 last_normality 0.303 OSMOL 0.276 Prealbumin PREALBUMIN 0.591 ALB 0.064 HCT 0.056 iSTAT Cg4 Pulse 0.273 BP_Low_Diastolic 0.176 Temp 0.129 iSTAT Creatinine CR 0.571 Birth 0.172 BP_High_Systolic 0.166 iSTAT Troponin I Comorbidity.MI 0.376 last_normality 0.161 TNI 0.155 Phosphorus last_normality 0.472 PHOS 0.153 CR 0.066 Platelet Count PLT 0.719 HCT 0.061 Temp 0.031 Procalcitonin CR 0.19 last_normality 0.139 PROCTL 0.101 Prothrombin Time PT 0.348 INR 0.262 last_normality 0.15 Teg PO2V 0.237 PCO2V 0.201 PO2A 0.156 Partial Thromboplastin Time last_normality 0.756 PTT 0.077 Pulse 0.032 Respiratory Culture last_normality 0.351 Urine 0.161 PCO2A 0.117 Respiratory Culture And Gram last_normality 0.359 Glasgow Coma 0.131 Resp 0.111 Stain Scale Score © 2019 Xu S et al. JAMA Network Open. Reticulocyte Count Automated last_normality 0.487 HCT 0.254 TBIL 0.076 Sepsis Protocol Lactate LACWBL 0.3 last_normality 0.125 Temp 0.103 Stool Culture BP_Low_Diastolic 0.161 Temp 0.108 Pulse 0.092 Occult Bld K 0.476 order_time 0.256 WBC 0.051 Troponin I TNI 0.442 last_normality 0.29 Pulse 0.045 Transferrin Saturation ALB 0.241 HCT 0.14 PLT 0.125 Triglycerides TGL 0.603 last_normality 0.244 Birth 0.017 Thyroid Stimulating Hormone last_normality 0.181 BP_High_Systolic 0.143 BP_Low_Diastolic 0.131 Osmolality K 0.254 CO2 0.189 BUN 0.187 Urinalysis With Microscopic Pulse 0.242 LABUA 0.142 AdmitDxDate 0.136 Urinalysis Pulse 0.128 BP_High_Systolic 0.125 Temp 0.095 Pregnancy Test AdmitDxDate 0.177 Pulse 0.167 BP_Low_Diastolic 0.155 Uric Acid URIC 0.26 last_normality 0.238 CR 0.161 Urine Culture Male 0.241 last_normality 0.142 Temp 0.131 Specific Gravity SPG 0.2 HCT 0.2 Temp 0.2 eTable 11: Top 3 Important Features for Stanford Standalone Labs lab feature 1 score feature 2 score feature 3 score 3 1 2 White Blood Cells WBC 0.558 last_normality 0.244 PLT 0.065 Hemoglobin HCT 0.287 HGB 0.217 last_normality 0.181 Platelet Count last_normality 0.665 PLT 0.238 Pulse 0.014 Sodium last_normality 0.787 NA 0.184 Resp 0.009 Potassium last_normality 0.576 K 0.271 Pulse 0.067 Creatinine last_normality 0.87 CR 0.1 Birth 0.005 Urea Nitrogen last_normality 0.809 BUN 0.089 CR 0.017 CO2 CO2 0.539 last_normality 0.141 Pulse 0.053 Calcium CA 0.532 last_normality 0.246 Pulse 0.042 Protein TP 0.677 last_normality 0.117 CA 0.033 Albumin ALB 0.459 last_normality 0.208 Pulse 0.107 © 2019 Xu S et al. JAMA Network Open. Alk Phos ALKP 0.672 last_normality 0.235 AdmitDxDate 0.016 Total Bilirubin TBIL 0.62 last_normality 0.307 PLT 0.019 AST (SGOT) AST 0.642 last_normality 0.217 Resp 0.028 ALT (SGPT) last_normality 0.848 ALT 0.115 ALB 0.008 eTable 12. Top 3 Important Features for Common Stanford Components lab feature 1 score feature 2 score feature 3 score 3 1 2 Magnesium last_normality 0.605 MAG 0.286 AdmitDxDate 0.033 Phosphorus PHOS 0.53 last_normality 0.154 CREAT 0.088 Hemoglobin A1c AdmitDxDate 0.171 order_time 0.141 Birth 0.099 Uric Acid last_normality 0.643 URIC 0.241 HCT 0.032 Albumin ALB 0.365 HCT 0.157 CAL 0.109 Thyroid Stimulating AdmitDxDate 0.173 order_time 0.139 Birth 0.122 Hormone Troponin I TROP 0.683 last_normality 0.151 AdmitDxDate 0.035 Potassium POT 0.364 Birth 0.116 AdmitDxDate 0.11 Sodium SOD 0.714 last_normality 0.082 POT 0.057 Calcium CAL 0.505 WBC 0.17 CREAT 0.105 eTable 13. Top 3 Important Features for Top UMich Standalone Labs lab feature 1 score feature 2 score feature 3 score 3 1 2 White Blood Cells last_normality 0.62 WBC 0.217 AdmitDxDate 0.048 Hemoglobin HGB 0.39 last_normality 0.338 HCT 0.209 Platelet Count last_normality 0.836 PLT 0.119 Birth 0.016 Sodium last_normality 0.837 SOD 0.11 AdmitDxDate 0.035 Potassium POT 0.667 last_normality 0.16 CREAT 0.063 Creatinine last_normality 0.881 CREAT 0.071 AdmitDxDate 0.017 © 2019 Xu S et al. JAMA Network Open. Total Bilirubin TBIL 0.475 last_normality 0.211 PLT 0.109 CO2 last_normality 0.755 CO2 0.174 AdmitDxDate 0.026 AST (SGOT) last_normality 0.753 AST 0.176 Birth 0.026 ALT (SGPT) last_normality 0.853 ALT 0.118 Birth 0.011 Albumin last_normality 0.802 ALB 0.074 Birth 0.054 Calcium CAL 0.622 last_normality 0.266 AdmitDxDate 0.021 Protein PROT 0.446 last_normality 0.222 HCT 0.059 Alk Phos last_normality 0.775 ALK 0.113 Birth 0.044 Urea Nitrogen last_normality 0.79 UN 0.116 Birth 0.034 eTable 14. Top 3 Important Features for Common UMich Components lab feature 1 score feature 2 score feature 3 score 3 1 2 Magnesium last_normality 0.585 MG 0.297 LABMGN 0.044 Phosphorus PO4 0.483 last_normality 0.169 CREAT 0.098 Prothrombin Time PT 0.46 INR 0.221 last_normality 0.152 Partial Thromboplastin Time PTT 0.593 last_normality 0.127 LABPTT 0.091 Alkaline Phosphatase ALKP 0.643 last_normality 0.183 Alkaline Phosphatase 0.038 Sodium NA 0.941 Pulse 0.015 last_normality 0.012 Potassium K 0.558 last_normality 0.168 LABK 0.067 Troponin I TRPI 0.302 last_normality 0.25 LABTNI 0.222 Lactate Dehydrogenase LD 0.595 last_normality 0.193 LABLDH 0.1 Calcium, Ionized CAI 0.529 last_normality 0.191 CA 0.172 Uric Acid URIC 0.629 last_normality 0.217 PLT 0.02 Albumin ALB 0.358 CA 0.187 HCT 0.107 Thyroid Stimulating Hormone last_normality 0.247 HCT 0.205 Temp 0.117 eTable 15. Top 3 Important Features for Top UCSF Standalone Labs © 2019 Xu S et al. JAMA Network Open. lab feature 1 score feature 2 score feature 3 score 3 1 2 White Blood Cells last_normality 0.719 WBC 0.216 Pulse 0.024 Hemoglobin last_normality 0.767 HGB 0.128 DBP 0.025 Platelet Count last_normality 0.886 PLT 0.091 SBP 0.007 Sodium last_normality 0.755 NA 0.191 CREAT 0.012 Potassium K 0.442 last_normality 0.118 SBP 0.072 Creatinine last_normality 0.836 CREAT 0.088 Comorbidity.RenalDisease 0.034 Total Bilirubin last_normality 0.791 TBILI 0.138 SBP 0.014 CO2 CO2 0.684 last_normality 0.218 HCT 0.025 AST (SGOT) last_normality 0.715 AST 0.196 Pulse 0.019 ALT (SGPT) ALT 0.587 last_normality 0.176 Pulse 0.074 Albumin ALB 0.502 last_normality 0.211 CA 0.078 Calcium CA 0.734 last_normality 0.125 Pulse 0.031 Protein TP 0.656 last_normality 0.137 DBP 0.048 Alk Phos last_normality 0.765 ALKP 0.175 SBP 0.01 Urea Nitrogen last_normality 0.745 BUN 0.139 Comorbidity.RenalDisease 0.018 eTable 16. Top 3 Important Features for Common UCSF Components © 2019 Xu S et al. JAMA Network Open. eMethods. Technical Details of Machine Learning Algorithm 1.Recursive feature elimination with cross validation (using a RandomForest estimator) We apply recursive feature elimination with cross validation (RFECV) to select top 5% relevant features for prediction as implemented by scikit-learn (http://scikit-learn.org/). This process uses all features in the development set to train a random forest prediction model, identifying which features appear the least important towards the predicted outcome as assessed by a Gini entropy score. We removed those least relevant features and repeated this process recursively, until there were only 5% of the total original features left. 2. Hyperparameter tuning for machine learning algorithms Eac that can tune the performance of the algorithms. For example, penalized logistic regression includes a regularization penalty that specifies how much the algorithm should balance using more information vs. developing the simplest model that uses the fewest number of features. Similarly, random forest models have a maximum tree depth hyperparameter that constrains how many features are considered when building decision trees, to balance the bias-variance tradeoff that can impact the generalizability of model accuracy. We systematically tested all models across a range of plausible hyperparameter values to identify the most effective choices (eTable 1). 3. Applying locally trained model to a remote dataset We apply pre-trained models to data of a remote site by manually mapping identical columns. to process the full UMich feature matrix with 603 columns. The template includes 43 columns imputed and selected (by RFECV) from Stanford training set. The same set of 43 features were then select from the UMich matrix and imputed accordingly if they exist. If instead, the UMich dataset does not have a corresponding feature (e.g. patient vitals), we will create a dummy column with the same feature name but filled with the corresponding constant imputation value. - ready to be fed into Stanford-trained model. © 2019 Xu S et al. JAMA Network Open. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JAMA Network Open American Medical Association

Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests

Loading next page...
 
/lp/american-medical-association/prevalence-and-predictability-of-low-yield-inpatient-laboratory-hRZLqFjriJ

References (50)

Publisher
American Medical Association
Copyright
Copyright 2019 Xu S et al. JAMA Network Open.
eISSN
2574-3805
DOI
10.1001/jamanetworkopen.2019.10967
Publisher site
See Article on Publisher Site

Abstract

Key Points Question How prevalent are low-yield IMPORTANCE Laboratory testing is an important target for high-value care initiatives, constituting inpatient diagnostic laboratory tests for the highest volume of medical procedures. Prior studies have found that up to half of all inpatient which results are predictable with laboratory tests may be medically unnecessary, but a systematic method to identify these machine learning models? unnecessary tests in individual cases is lacking. Findings In this diagnostic study of 191 506 inpatients from 3 tertiary OBJECTIVE To systematically identify low-yield inpatient laboratory testing through personalized academic medical centers, common predictions. low-yield inpatient diagnostic laboratory test results were systematically DESIGN, SETTING, AND PARTICIPANTS In this retrospective diagnostic study with multivariable identified through data-driven methods prediction models, 116 637 inpatients treated at Stanford University Hospital from January 1, 2008, and personalized predictions. to December 31, 2017, a total of 60 929 inpatients treated at University of Michigan from January 1, 2015, to December 31, 2018, and 13 940 inpatients treated at the University of California, San Meaning The findings suggest that Francisco from January 1 to December 31, 2018, were assessed. data-driven methods can make explicit the level of uncertainty and expected MAIN OUTCOMES AND MEASURES Diagnostic accuracy measures, including sensitivity, specificity, information gain from diagnostic tests, negative predictive values (NPVs), positive predictive values (PPVs), and area under the receiver with the potential to encourage useful operating characteristic curve (AUROC), of machine learning models when predicting whether testing and discourage low-value testing inpatient laboratory tests yield a normal result as defined by local laboratory reference ranges. that can incur direct cost and indirect harm. RESULTS In the recent data sets (July 1, 2014, to June 30, 2017) from Stanford University Hospital (including 22 664 female inpatients with a mean [SD] age of 58.8 [19.0] years and 22 016 male Supplemental content inpatients with a mean [SD] age of 59.0 [18.1] years), among the top 20 highest-volume tests, 792 397 were repeats of orders within 24 hours, including tests that are physiologically unlikely to Author affiliations and article information are listed at the end of this article. yield new information that quickly (eg, white blood cell differential, glycated hemoglobin, and serum albumin level). The best-performing machine learning models predicted normal results with an AUROC of 0.90 or greater for 12 stand-alone laboratory tests (eg, sodium AUROC, 0.92 [95% CI, 0.91-0.93]; sensitivity, 98%; specificity, 35%; PPV, 66%; NPV, 93%; lactate dehydrogenase AUROC, 0.93 [95% CI, 0.93-0.94]; sensitivity, 96%; specificity, 65%; PPV, 71%; NPV, 95%; and troponin I AUROC, 0.92 [95% CI, 0.91-0.93]; sensitivity, 88%; specificity, 79%; PPV, 67%; NPV, 93%) and 10 common laboratory test components (eg, hemoglobin AUROC, 0.94 [95% CI, 0.92-0.95]; sensitivity, 99%; specificity, 17%; PPV, 90%; NPV, 81%; creatinine AUROC, 0.96 [95% CI, 0.96-0.97]; sensitivity, 93%; specificity, 83%; PPV, 79%; NPV, 94%; and urea nitrogen AUROC, 0.95 [95% CI, 0.94, 0.96]; sensitivity, 87%; specificity, 89%; PPV, 77%; NPV 94%). CONCLUSIONS AND RELEVANCE The findings suggest that low-yield diagnostic testing is common and can be systematically identified through data-driven methods and patient context–aware predictions. Implementing machine learning models appear to be able to quantify the level of uncertainty and expected information gained from diagnostic tests explicitly, with the potential to (continued) Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 1/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Abstract (continued) encourage useful testing and discourage low-value testing that incurs direct costs and indirect harms. JAMA Network Open. 2019;2(9):e1910967. Corrected on October 11, 2019. doi:10.1001/jamanetworkopen.2019.10967 Introduction 1,2 Unsustainable growth in health care costs is exacerbated by waste that does not improve health. The Institute of Medicine estimates that more than $200 billion a year is spent on unnecessary tests and procedures. Given this amount of misallocated resources, there has been an increasing emphasis on high-value care, notably with the American Board of Internal Medicine Foundation’s Choosing Wisely guidelines. Laboratory testing, in particular, constitutes the highest-volume medical procedure, with estimates of up to 25% to 50% of all inpatient testing being medically 6,7 unnecessary. The consequences of unnecessary testing are not simply financial but also include low patient satisfaction, sleep fragmentation, risk of delirium, iatrogenic anemia, and increased 8-11 mortality. Numerous interventions have been studied to reduce inappropriate laboratory testing, including clinical education, audit feedback, financial incentives, and electronic medical record 12-15 (EMR)–based ordering restrictions. Interventions based on EMRs offer pertinent information for clinical decision-making, such as cost, turnaround time, prior stable results, and guideline-based best 16-20 practice alerts. Despite these efforts, unnecessary tests remain prolific when practitioners are influenced by fear of missing problems, medicolegal concerns, patient preferences, and the overall difficulty of systematically identifying low-value testing at the point of care, prompting behavior to 21,22 check just in case. We envisioned patient-specific estimates of the pretest probability of results for any diagnostic test, displayed at the point of clinical order entry. When humans tend to have poor intuition for estimating probabilities and diagnostic test performance, having automated computer systems explicitly provide those estimates could substantially change clinical practice. Machine learning in medicine now offers a direct mechanism to produce such estimates by predicting select laboratory 24-30 results. Although prior approaches can provide a laboratory result given other simultaneously available results (eg, estimating ferritin levels when other components of an iron panel are given), this is too late for decision support to change behavior when the tests are already performed. We addressed the more clinically relevant question of predicting laboratory results with only information available before the test is ordered. Our objective was to identify inpatient diagnostic laboratory testing with predictable results that are unlikely to yield new information. Our analytic approach escalated from descriptive statistics to machine learning models for individualized estimates of predictable test results. Methods This diagnostic study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline for reporting results of multivariate prediction models to develop and evaluate our machine learning methods (eFigure 1 in the Supplement gives an overview of our approach). Ten years (January 1, 2008, to December 31, 2017) of inpatient electronic medical record (EMR) data from hospitals at Stanford University, 4 years (January 1, 2015, to December 31, 2018) of data from University of Michigan (UMich), and 1 year (2018) of data from University of California, San Francisco (UCSF) were used for this study. To preserve data privacy, raw clinical data were deidentified, processed, trained, and evaluated locally at each local site, with only evaluation results sent back to Stanford for further analysis. The Stanford University, UMich, and UCSF JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 2/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests institutional review boards approved the study at each site. Project-specific informed consent was not required because the study was restricted to secondary analysis of existing clinical data. Patient data at Stanford University were extracted and deidentified by the STRIDE (Stanford Translational Research Integrated Database Environment) project, a research and development project at Stanford University to create a standards-based informatics platform supporting clinical and translational research. Participants and Inclusion Criteria All laboratory test results had reference labels for normal vs abnormal results as defined by local clinical laboratory reference ranges and at least 500 occurrences in a data set. For each laboratory test, we retrieved a random sample of 10 000 test orders from the available data (or all orders if <10 000). Outcome and Evaluation Metrics Our goal was to predict the result (negative vs positive) of each laboratory test using information available before the order was placed. We considered stand-alone tests, in which a single order yielded a single result (eg, magnesium level, lactate level, or blood cultures), and panel tests that yielded multiple component results (eg, a complete blood cell count panel yielded white blood cell, hemoglobin, and platelet component results). We predicted the results of each panel component separately to avoid labeling an entire panel as positive or negative. We evaluated prediction performance through standard metrics for diagnostic accuracy, including the area under the receiver operating characteristic curve (ie, AUROC or C statistic), which summarizes the trade-off between sensitivity and specificity. Given specific decision thresholds, we calculated diagnostic test metrics, including sensitivity, specificity, positive predictive value, and negative predictive value (NPV). Typically, such metrics evaluate how well a test predicts a diagnosis. In our case, a test result being abnormal was itself the diagnosis, whereas the prediction algorithms operated as screening tests compared with the physical laboratory tests. For example, NPV was the probability of being correct when a negative or normal result was predicted. Predictors and Data Feature For each laboratory, 875 raw features from the Stanford University EMR that reflected patient clinical context available at the time of the order entry were extracted (eTable 1 in the Supplement). The core features included patient demographics, normality of the most recent test of interest, numbers of recent tests of interest, history of Charlson Comorbidity Index categories, which specialty team was treating the patient, time since admission, time of day and year of the test, and summary statistics of recent vital statistics and laboratory results. Vital statistics and treatment team information were not accessible in the UMich data sample, which yielded 603 raw features. Age and sex information were not accessible in the UCSF data sample, which yielded 806 raw features. Development vs Validation Split Patients were randomly split into training (development) and held-out test (validation) sets with a 75:25 split. The model was developed based on the training data alone but assessed generalizable predictive accuracy on the separate patients in the held-out sets. Missing Data Most of the data features, such as history of a comorbidity category or the number of prior laboratory tests, always had a valid value (including not present or zero). Numerical results (eg, mean sodium level in the past week) could be missing, in which case we carried forward the most recent value from the patient’s prior records. If no prior values existed, we imputed the training sample mean. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 3/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Feature Selection We applied recursive feature elimination (with cross-validation) to select the top 5% most important features for model building that best improved accuracy when included in prediction models. This resulted in 43 processed features in each subsequent prediction model (the eMethods in the Supplement gives technical explanations). Model Development We built an array of prediction models using established algorithms, including regularized logistic regression, regress and round, naive Bayes, neural network multilayer perceptrons, decision tree, random forest, AdaBoost, and XGBoost. Each model generated a prediction score between 0 and 1 for how likely a laboratory test result would be negative or normal vs positive or abnormal (Figure 1A). A baseline model predicted the most recent result (if the patient had a prior test) or the Figure 1. Normality Scores, Decision Threshold, and Receiver Operating Characteristic (ROC) Curve A Before thresholding B After thresholding 800 800 Abnormal False negative Normal True positive True negative 600 600 False positive 400 400 200 200 0 0 0 0.2 0.4 0.6 0.8 1.0 0 0.2 0.4 0.6 0.8 1.0 Score Score C Threshold on ROC 1.0 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1.0 1–Specificity A, Histogram of normality scores distributed among normal and abnormal orders. B, false-positive orders were predicted to be abnormal but actually were normal, and true- After picking a threshold, orders were classified as predicted to be normal if their positive orders were predicted to be abnormal but actually were abnormal. C, This choice normality scores were above the threshold or predicted to be abnormal if they were of threshold led to a sensitivity of 96% and specificity of 67%, as shown on the below the threshold. True-negative orders were predicted to be normal but actually were ROC curve. normal, false-negative orders were predicted to be normal but actually were abnormal, JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 4/13 Sensitivity No. of Orders No. of Orders JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests overall prevalence of positive results as the prediction score. Additional model specifications are included in eTable 2 in the Supplement. Decision Threshold Estimation Decision thresholds translate continuous prediction scores into discrete negative vs positive predictions (Figure 1B). We conservatively favored high sensitivity and high NPV to minimize the risks 35,36 of alert fatigue and missing clinically important laboratory test result abnormalities (Figure 1C). This article gives results targeting an NPV of 95%, recognizing the diminishing returns of expected information gain when one is already 95% certain of the result. These diminishing returns were easily adjusted for different clinical scenarios with varying tolerances for uncertainty because we confirmed robustness across a range of options targeting NPVs of 99%, 95%, 90%, and 80% (eTable 3 in the Supplement). Statistical Analysis To assess the statistical significance of the results, we calculated 95% CIs for AUROCs by resampling the evaluation set 1000 times for each laboratory (Table and eTables 3-8 in the Supplement). We performed additional randomized permutation tests to compare the AUROC of the best-performing algorithm against that of the baseline model (eFigures 2-7 in the Supplement). Multisite Evaluation We performed equivalent analysis from multiple sites, including hospitals at Stanford University, UMich, and UCSF. We developed mapping software between the data formats from different sites to allow for a common analytic process at each site without sharing raw clinical data. We cross- evaluated performances of models trained at one site and then tested at another. Results Prevalence of Repetitive Tests The recent data sets (July 1, 2014, to June 30, 2017) from Stanford University Hospital included 22 664 female inpatients (mean [SD] age, 58.8 [19.0] years) and 22 016 male inpatients (mean [SD] Table. Diagnostic Performance Metrics of Top-Volume Stand-Alone Laboratory Tests Predicting Whether Laboratory Tests Will Yield a Normal Result on a Held-Out Evaluation Set, Targeting at an NPV of 95% No. of Orders/ Metric, % 1000 Patient Laboratory Test Encounters AUROC (95% CI) Prevalence NPV PPV Sensitivity Specificity TN FN TP FP Magnesium 4246 0.76 (0.74-0.78) 26 91 36 86 47 35 3.6 22 39 Prothrombin time 2244 0.89 (0.88-0.91) 80 85 81 100 3.6 0.7 0.1 80 19 Phosphorus 2120 0.74 (0.72-0.76) 33 88 39 91 30 20 2.8 30 47 Partial thromboplastin 1471 0.86 (0.85-0.87) 61 87 65 98 17 6.5 1.0 60 32 time Lactate 1230 0.87 (0.85-0.88) 29 91 56 82 74 53 5.2 23 19 Calcium, ionized 1197 0.72 (0.70-0.74) 61 90 62 100 4.8 1.9 0.2 61 37 Potassium 752 0.81 (0.79-0.84) 12 92 40 43 91 80 7.0 5.2 7.9 Troponin I 534 0.92 (0.91-0.93) 33 93 67 88 79 53 4.0 29 14 LDH 455 0.93 (0.93-0.94) 47 95 71 96 65 35 1.8 45 18 Blood culture Aerobic and anaerobic 400 0.66 (0.61-0.71) 8.1 93 16 16 93 85 6.8 1.3 6.6 2 Aerobic 371 0.62 (0.58-0.67) 9.1 93 12 61 54 49 3.6 5.6 42 Sodium 361 0.92 (0.91-0.93) 57 93 66 98 35 15 1.1 56 28 Abbreviations: AUROC, area under the receiver operating characteristic curve; FN, false- Prevalence of abnormal or positive test results. negative; FP, false-positive; LDH, lactate dehydrogenase; NPV, negative predictive value; PPV, positive predictive value; TN, true-negative; TP, true-positive. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 5/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests age, 59.0 [18.1] years). Figure 2A reports the overall volume of the most commonly repeated inpatient laboratory tests at Stanford University during that period. Among the top 20 volume tests, 792 397 were repeats of orders within 24 hours. Figure 2B reports the repetition rate of common tests medically implausible to yield new information from frequent testing (eg, glycated hemoglobin). The likelihood of common laboratory components that yielded a negative result progressively increased as repeated negative results were observed (Figure 2C). Model Performance Random forest and XGBoost demonstrated the highest discriminating power for most of the stand- alone laboratory tests from Stanford University hospital, yielding a mean AUROC of 0.77 compared with 0.67 with the baseline model (eFigure 2 in the Supplement gives the ROC curves). The best- performing machine learning models predicted normal results with an AUROC of 0.90 or greater for 12 stand-alone laboratory tests (eg, sodium AUROC, 0.92 [95% CI, 0.91-0.93]; sensitivity, 98%; specificity, 35%; PPV, 66%; NPV, 93%; lactate dehydrogenase AUROC, 0.93 [95% CI, 0.93-0.94]; sensitivity, 96%; specificity, 65%; PPV, 71%; NPV, 95%; and troponin I AUROC, 0.92 [95% CI, 0.91- 0.93]; sensitivity, 88%; specificity, 79%; PPV, 67%; NPV, 93%) and 10 common laboratory test components (eg, hemoglobin AUROC, 0.94 [95% CI, 0.92-0.95]; sensitivity, 99%; specificity, 17%; PPV, 90%; NPV, 81%; creatinine AUROC, 0.96 [95% CI, 0.96-0.97]; sensitivity, 93%; specificity, 83%; PPV, 79%; NPV, 94%; and urea nitrogen AUROC, 0.95 [95% CI, 0.94, 0.96]; sensitivity, 87%; specificity, 89%; PPV, 77%; NPV 94%). Diagnostic performance metrics for the most common stand- alone laboratory tests when targeting 95% NPV are given in the Table, with the full table of all Figure 2. Prevalence of Repetitive Tests and Their Diminishing Information Gain <24 h 24 h to 3 d 3 d to 1 wk >1 wk A B Highly repeated common tests Guideline-controlled tests C-peptide Basic metabolic panel Albumin Magnesium Lipase CBC with diff Aspergillus galactomannan Prothrombin time C-reactive protein Phosphorus Glycated hemoglobin PTT NT-proBNP Lactate Sedimentation rate Calcium ionized Thyrotropin Potassium 0 20 40 60 80 100 Troponin I LDH total Tests, % Heparin activity level Sodium C Consecutive normal test findings in past 7 days Lidocaine level Hematocrit 1.0 Uric acid 0.8 Sepsis protocol lactate Lactic acid White blood cells 0.6 Fibrinogen Hemoglobin Specific gravity 0.4 Sodium Potassium 0 1000 2000 3000 4000 5000 0.2 Creatinine No. of Orders per 1000 Patient Encounters 0 1 2 3 4 5 Consecutive Normal Test Findings in Past 7 Days A, Most commonly repeated laboratory test orders from July 1, 2014, to June 30, 2017, and 6.7% of glycated hemoglobin inpatient tests were performed again within 24 hours, at Stanford University Hospital. The total length of each bar represents the total volume even when it was not biologically plausible for the results to meaningfully change that of laboratory orders per 1000 patient encounters, with shaded regions reflecting how rapidly. C, Prevalence of normal results for common laboratory components many of these were repeated orders within a given time. For example, 47% of basic progressively increased toward 100% as more subsequent normal results were observed metabolic panels were subsequent tests performed again within 24 hours of the past in the prior week. CBC with diff indicates complete blood cell count with differential; order. Results were sorted by this number of repeated tests within 24 hours. B, LDH, lactate dehydrogenase; NT-proBNP, N-terminal pro–brain-type natriuretic peptide; Distribution of repeated orders for laboratory tests specifically identified as rarely ever and PTT, partial thromboplastin time. having clinical justification for repeated daily testing. For example, 18.3% of albumin JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 6/13 Normal Rate JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests laboratory tests evaluated in eTable 3 in the Supplement. Performance metrics for the common components in complete blood cell counts and comprehensive metabolic panels are given in Figure 3 along with results from UMich and UCSF data, with the full table of diagnostic performance metrics in eTable 4 in the Supplement and ROC curves in eFigure 3 in the Supplement. Model Transferability The respective prediction results for UMich data are reported in eFigure 4, eFigure 5, eTable 5, and eTable 6 in the Supplement, whereas similar results from UCSF are reported in eFigure 6, eFigure 7, eTable 7, and eTable 8 in the Supplement. Figure 4 gives the performance of models trained at Stanford University and subsequently evaluated at all sites. Although cross-site performance declined compared with local performance (eg, when predicting albumin results, AUROC decreased from 0.92 [95% CI, 0.91-0.94] when locally tested at Stanford University to 0.73 [95% CI, 0.70- 0.75] when remotely tested at UMich), predictive power was retained (AUROC, >0.85) for most Figure 3. Diagnostic Metrics of Predictions on Common Components in the Data Sets of Stanford University, University of Michigan (UMich), and University of California, San Francisco (UCSF) False positives True positives True negatives False negatives Fractions of true-negative, false-negative, false- Stanford UMich UCSF positive, and true-positive results are scaled by the Predicted Predicted Predicted Predicted Predicted Predicted number of orders among each 1000 patient Abnormal Normal Abnormal Normal Abnormal Normal encounters. Predicted normal represents the volume White blood cells that the model would suggest not to order, and we Hemoglobin targeted to limit the fraction of false-negative results Platelets Sodium to less than 5%. For some laboratory tests (eg, albumin Potassium measurement at Stanford University), there were Carbon dioxide almost zero predicted normal results, which means Urea nitrogen Creatinine that a few orders existed in the training set that were Calcium unpredictable; thus, the predictor could not Albumin confidently achieve a 95% negative predictive value Protein Alkaline phosphatase by picking any threshold above 0. The model chose a Total bilirubin decision threshold equal to 0, which led to scores of all AST orders in the test set falling above the decision ALT threshold, thus always encouraging ordering the test. 8000 6000 3000 0 3000 6000 6000 3000 0 3000 6000 6000 3000 0 3000 6000 ALT indicates alanine aminotransferase; AST, aspartate No. of Orders per No. of Orders per No. of Orders per 1000 Patient Encounters 1000 Patient Encounters 1000 Patient Encounters aminotransferase. Figure 4. Area Under the Receiver Operating Characteristic Curve (AUROC) Scores of Models for 15 Common Laboratory Test Components Developed at Stanford University but Evaluated at All Sites White blood cells UMich Hemoglobin UCSF Platelets Stanford Sodium Potassium Carbon dioxide Urea nitrogen Creatinine Calcium Albumin Protein The model generally achieved highest performance Alkaline phosphatase when evaluated locally at Stanford University with an Total bilirubin AUROC of 0.9 or greater for 10 laboratory test AST components but still retained at 0.85 or greater in 9 ALT cases when evaluated remotely at University of 0.5 0.6 0.7 0.8 0.9 1.0 California, San Francisco (UCSF) and University of AUROC Score Michigan (UMich). JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 7/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests laboratory components (eTable 9 in the Supplement gives the full comparison data). For certain tests, such as sodium level, however, the model trained at Stanford University had a better AUROC when tested at UMich (0.91; 95% CI, 0.90-0.93) than locally at Stanford University (0.87; 95% CI, 0.85-0.88). Inspection of the data and model showed that the UMich sodium level was easier to predict, with a baseline model already yielding an AUROC of 0.87 at UMich and 0.79 at Stanford University. Discussion Interpretation This study systematically identified low-yield diagnostic laboratory tests. Starting with simple descriptive statistics, Figure 2 shows how frequently laboratory tests are performed again. Although some tests may have credible reasons for such frequent repetition, guidelines and external knowledge can help identify some low-value repeated tests. For example, hundreds of tests for serum albumin, thyrotropin, and glycated hemoglobin levels were performed again within 24 hours, along with tens of thousands of repetitive tests for phosphorus and complete blood cell counts with differential. This finding quantitatively supports issues suggested in previous guidelines that hospitals can immediately use to target unnecessary repeated tests, such as through best practice 12,19,38-40 alerts showing recently available test results. Most instances of low-yield testing are not as straightforward to identify; thus, our study added machine learning methods for personalized test result predictions. Additional features, such as patient demographics, vital signs, and other common laboratory results, can be synthesized through machine learning models to produce more robust and accurate predictions. Although different applications and clinical contexts will have different tolerances for uncertainty, the study gave the primary results when choosing a conservative target NPV close to 95% (when the model predicted a test result was going to be normal, the goal was for it to be correct 95% of the time). This approach fits a scenario in which these targets are implemented as best practice alerts with a desire to maintain a small number of false-positive results (5%). The results at this level of pretest were estimated by which pursuing further testing would yield markedly diminishing returns. eTables 3 through 8 in the Supplement give similar results across a range of different NPV targets. Consistent with existing guideline-based forms of clinical decision support, pretest estimates of whether a laboratory test result will be normal would inform physician decision-making but not dictate or replace it. Ultimately, medical testing decisions are always based on varying levels of diagnostic certainty, even if practitioners are only implicitly aware that they are empirically estimating probabilistic risks based on patient characteristics. For example, blood cultures are not performed for every febrile patient because a credible risk of bacteremia is qualitatively recognized in only certain situations. Likewise, blood cultures are performed in sets of 4 bottles at a time, but we do not continue to check 5 or more bottles because we recognize further repeated tests are unlikely to yield information that was not already predictable based on the prior results. This approach provides a systematic and quantitative way to inform such decisions. The results should encourage practitioners and quality improvement committees to make explicit and quantitative their own embedded assumptions on acceptable decision thresholds. The general framework presented to quantify uncertainty can then feed into individual point-of-care decisions or more formal decision analyses. Implications This study provides a general approach to identifying predictable laboratory tests. Many of the laboratory tests that we evaluated have been evaluated for overuse, including magnesium 15,43-45 46 47 level, blood cultures, and complete blood cell counts. Patient-specific estimates of laboratory test result normality at the point-of-order entry may discourage low-yield tests with predictably negative results and encourage appropriate tests with high levels of uncertainty. For JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 8/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests example, when our method did not predict a blood culture result to be negative, this corresponds to greater than 16% positive predictive value (Table). This finding is more than enough risk of bacteremia to prompt diagnostic testing and even empirical treatment. This approach can also raise questions on how guideline- and protocol-based testing is implemented and could be optimized. The optimal threshold of acceptable uncertainty depends on the clinical scenario and the particular test. For example, although screening tests (eg, HIV testing or pregnancy screens in hospital settings) have predictable normal results, most of the time, they are unlikely to be influenced by decision support when the effect of missing an abnormal case is sufficiently severe and driven by overriding protocols. Similarly, regulatory requirements around sepsis protocols are a major driver of repeated lactate testing that may not be amenable to decision support on predictable results. The results of this study can still inform the development of such regulatory requirements on the appropriate number and interval of screening tests that may otherwise be excessive or too rigid for individual cases. In predictable cases, the risk of false-positive test results (and adverse downstream effects) may be substantial. These results can also provide foundational quantitative support for cost-effectiveness analysis. For example, if scaling the annual volume of predictable tests (predicted normal results) by their financial costs (eTable 10 in the Supplement), one could estimate annual savings by avoiding these tests. However, this saving should be carefully compared against potential harms and costs generated from missing the actually abnormal tests (false-negative results). In cases of panel test ordering, practitioners are often only interested in 1 or 2 components of panel tests at a time (eg, sodium level from a metabolic panel or hemoglobin level from a complete blood cell count). Most panel components may be predictably normal, but there could still be value in the overall order if there is sufficient uncertainty in at least 1 other clinically relevant component. Our separate predictions for each panel component in Figure 3 would allow practitioners to decide which components are relevant for their decision-making in future point-of-care information displays. The results also allow us to systematically identify relevant factors that are predictive of each test result. This identification can inform simple rule-based clinical decision support based on factors including obvious elements, such as prior results, and less obvious ones, such as sex for ferritin status and surgical vs medical team for cerebrospinal fluid studies. eTables 11 through 16 in the Supplement include a full list of the most important features for predicting the normality of each laboratory test result. Limitations Although we used conservative fixed-decision thresholds for clarity (targeting 95% NPV) in this proof-of-concept study, specific applications can undergo explicit decision analysis to assess the balance between risk and benefit. Even then, such future studies would require the foundation that we have established to assess the relative likelihood of different testing outcomes. Assuming that the training data reflect the same distribution as the evaluation, intended application data distribution is an important limitation in any prediction model. Although we believe it may ultimately be more valuable to disseminate our underlying approach to undergo continuous learning and adaptation to local environments, we assessed model performance across multiple sites. Figure 4 shows that models trained at Stanford University can often still retain useful predictive performance when evaluated at UCSF and UMich, although these models will predictably underperform locally trained models. For example, the decrease in performance when predicting albumin levels at UMich with the model trained at Stanford University is likely associated with different underlying population distributions, including substantially different prevalences of normal albumin test results (16% at Stanford vs 57% at UMich). On the other hand, the surprising increase of AUROC when applying the sodium model trained at Stanford University to UMich may indicate that sodium level was more excessively tested at UMich, making it easier to identify predictable repeated tests in their data. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 9/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Another factor that may lead to prediction failure is that the distribution of data could change over time. The point of refining decision support systems is to change ordering behavior, which is itself one of the most useful inputs into the predictive models. Consequently, we would recommend online learning algorithms that continuously adapt to practice changes rather than ever expecting to have a completed final model. Conclusions The findings suggest that low-yield diagnostic testing is common and can be systematically identified through data-driven methods and patient context–aware predictions. Implementing continuous learning prediction models may help quantify the level of uncertainty and expected information gain from diagnostic tests explicitly, with potential to encourage useful testing and discourage low-value testing that can incur direct costs and indirect harms. ARTICLE INFORMATION Accepted for Publication: July 22, 2019. Published: September 11, 2019. doi:10.1001/jamanetworkopen.2019.10967 Correction: This article was corrected on October 11, 2019, to fix errors in Figure 2B, Figure 3, and Limitations. Open Access: This is an open access article distributed under the terms of the CC-BY License.©2019XuSetal. JAMA Network Open. Corresponding Author: Jonathan H. Chen, MD, PhD, Center for Biomedical Informatics Research, Department of Medicine, Stanford University, 1265 Welch Rd, Medical School Office Bldg X213, Stanford, CA 94305 (jonc101@ stanford.edu). Author Affiliations: Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California (Xu, Balasubramanian, Chen); Division of Hospital Medicine, Department of Medicine, Stanford University, Stanford, California (Hom, Chen); Department of Pathology, University of Michigan School of Medicine, Ann Arbor (Schroeder); Department of Medicine, University of California, San Francisco (Najafi); Department of Computer Science, Stanford University, Stanford, California (Roy). Author Contributions: Drs Xu and Hom contributed equally to this work. Drs Xu and Chen had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: Roy, Chen. Acquisition, analysis, or interpretation of data: Xu, Hom, Balasubramanian, Schroeder, Najafi, Chen. Drafting of the manuscript: Xu, Hom, Najafi, Roy, Chen. Critical revision of the manuscript for important intellectual content: Xu, Hom, Balasubramanian, Schroeder, Chen. Statistical analysis: Xu, Balasubramanian, Chen. Obtained funding: Chen. Administrative, technical, or material support: Balasubramanian, Schroeder, Chen. Supervision: Chen. Conflict of Interest Disclosures: Dr Chen reported receiving grants from the National Institute of Environmental Health Sciences and the Gordon and Betty Moore Foundation during the conduct of the study and having co-ownership of Reaction Explorer LLC (chemistry education software company). No other disclosures were reported. Funding/Support: This study was supported by National Institutes of Health Big Data 2 Knowledge Award K01ES026837 through the National Institute of Environmental Health Sciences and in part by grant GBMF8040 from the Gordon and Betty Moore Foundation (Dr Chen). The STRIDE project was supported by grant UL1 RR025744 from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health. Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 10/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or Stanford Health Care. REFERENCES 1. Alvin MD, Jaffe AS, Ziegelstein RC, Trost JC. Eliminating creatine kinase-myocardial band testing in suspected acute coronary syndrome: a value-based quality improvement. JAMA Intern Med. 2017;177(10):1508-1512. doi:10. 1001/jamainternmed.2017.3597 2. Dieleman JL, Squires E, Bui AL, et al. Factors associated with increases in US health care spending, 1996-2013. JAMA. 2017;318(17):1668-1678. doi:10.1001/jama.2017.15927 3. Smith M, Saunders R, Stuckhardt L, McGinnis JM, eds; Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: National Academies Press; 2014. 4. Colla CH, Morden NE, Sequist TD, Schpero WL, Rosenthal MB. Choosing Wisely: prevalence and correlates of low-value health care services in the United States. J Gen Intern Med. 2015;30(2):221-228. doi:10.1007/s11606- 014-3070-z 5. Zhi M, Ding EL, Theisen-Toupal J, Whelan J, Arnaout R. The landscape of inappropriate laboratory testing: a 15- year meta-analysis. PLoS One. 2013;8(11):e78962. doi:10.1371/journal.pone.0078962 6. Yeh DD. A clinician’s perspective on laboratory utilization management. Clin Chim Acta. 2014;427(January): 145-150. doi:10.1016/j.cca.2013.09.023 7. Konger RL, Ndekwe P, Jones G, et al. Reduction in unnecessary clinical laboratory testing through utilization management at a US government Veterans Affairs hospital. Am J Clin Pathol. 2016;145(3):355-364. doi:10.1093/ ajcp/aqv092 8. Koch CG, Li L, Sun Z, et al. Hospital-acquired anemia: prevalence, outcomes, and healthcare implications. JHosp Med. 2013;8(9):506-512. doi:10.1002/jhm.2061 9. Koch CG, Li L, Sun Z, et al. From bad to worse: anemia on admission and hospital-acquired anemia. J Patient Saf. 2017;13(4):211-216. doi:10.1097/PTS.0000000000000142 10. Thavendiranathan P, Bagai A, Ebidia A, Detsky AS, Choudhry NK. Do blood tests cause anemia in hospitalized patients? the effect of diagnostic phlebotomy on hemoglobin and hematocrit levels. J Gen Intern Med. 2005;20 (6):520-524. doi:10.1111/j.1525-1497.2005.0094.x 11. Salisbury AC, Reid KJ, Alexander KP, et al. Diagnostic blood loss from phlebotomy and hospital-acquired anemia during acute myocardial infarction. Arch Intern Med. 2011;171(18):1646-1653. doi:10.1001/archinternmed.2011.361 12. Eaton KP, Levy K, Soong C, et al. Evidence-based guidelines to eliminate repetitive laboratory testing. JAMA Intern Med. 2017;177(12):1833-1839. doi:10.1001/jamainternmed.2017.5152 13. Stammen LA, Stalmeijer RE, Paternotte E, et al. Training physicians to provide high-value, cost-conscious care: a systematic review. JAMA. 2015;314(22):2384-2400. doi:10.1001/jama.2015.16353 14. Hom J, Kumar A, Evans KH, et al. A high value care curriculum for interns: a description of curricular design, implementation and housestaff feedback. Postgrad Med J. 2017;93(1106):725-729. doi:10.1136/postgradmedj- 2016-134617 15. Han SJ, Saigal R, Rolston JD, et al. Targeted reduction in neurosurgical laboratory utilization: resident-led effort at a single academic institution. J Neurosurg. 2014;120(1):173-177. doi:10.3171/2013.8.JNS13512 16. Sedrak MS, Myers JS, Small DS, et al. Effect of a price transparency intervention in the electronic health record on clinician ordering of inpatient laboratory tests: the PRICE randomized clinical trial. JAMA Intern Med. 2017;177 (7):939-945. doi:10.1001/jamainternmed.2017.1144 17. Silvestri MT, Bongiovanni TR, Glover JG, Gross CP. Impact of price display on provider ordering: a systematic review. JHospMed. 2016;11(1):65-76. doi:10.1002/jhm.2500 18. Feldman LS, Shihab HM, Thiemann D, et al. Impact of providing fee data on laboratory test ordering: a controlled clinical trial. JAMA Intern Med. 2013;173(10):903-908. doi:10.1001/jamainternmed.2013.232 19. Jun T, Kwang H, Mou E, et al. An electronic best practice alert based on Choosing Wisely guidelines reduces thrombophilia testing in the outpatient setting. J Gen Intern Med. 2019;34(1):29-30. doi:10.1007/s11606-018- 4663-8 20. Felcher AH, Gold R, Mosen DM, Stoneburner AB. Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing. J Am Med Inform Assoc. 2017;24(4):776-780. doi: 10.1093/jamia/ocw182 21. Sedrak MS, Patel MS, Ziemba JB, et al. Residents’ self-report on why they order perceived unnecessary inpatient laboratory tests. JHospMed. 2016;11(12):869-872. doi:10.1002/jhm.2645 JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 11/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests 22. Miyakis S, Karamanof G, Liontos M, Mountokalakis TD. Factors contributing to inappropriate ordering of tests in an academic medical department and the effect of an educational feedback strategy. Postgrad Med J. 2006; 82(974):823-829. doi:10.1136/pgmj.2006.049551 23. Manrai AK, Bhatia G, Strymish J, Kohane IS, Jain SH. Medicine’s uncomfortable relationship with math: calculating positive predictive value. JAMA Intern Med. 2014;174(6):991-993. doi:10.1001/jamainternmed. 2014.1059 24. Luo Y, Szolovits P, Dighe AS, Baron JM. Using machine learning to predict laboratory test results. Am J Clin Pathol. 2016;145(6):778-788. doi:10.1093/ajcp/aqw064 25. Lidbury BA, Richardson AM, Badrick T. Assessment of machine-learning techniques on large pathology data sets to address assay redundancy in routine liver function test profiles. Diagnosis (Berl). 2015;2(1):41-51. doi:10. 1515/dx-2014-0063 26. Aikens RC, Balasubramanian S, Chen JH. A machine learning approach to predicting the stability of inpatient lab test results. AMIA Jt Summits Transl Sci Proc. 2019;2019:515-523. 27. Cabitza F, Banfi G. Machine learning in laboratory medicine: waiting for the flood? Clin Chem Lab Med. 2018; 56(4):516-524. doi:10.1515/cclm-2017-0287 28. Chen JH, Asch SM. Machine learning and prediction in medicine: beyond the peak of inflated expectations. N Engl J Med. 2017;376(26):2507-2509. doi:10.1056/NEJMp1702071 29. Obermeyer Z, Emanuel EJ. Predicting the future: big data, machine learning, and clinical medicine. N Engl J Med. 2016;375(13):1216-1219. doi:10.1056/NEJMp1606181 30. Beam AL, Kohane IS. Big data and machine learning in health care. JAMA. 2018;319(13):1317-1318. doi:10.1001/ jama.2017.18391 31. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162(1):55-63. doi:10. 7326/M14-0697 32. Hripcsak G. Visualizing the operating range of a classification system. J Am Med Inform Assoc. 2012;19(4): 529-532. doi:10.1136/amiajnl-2011-000633 33. Hastie T, Tibshirani R, Friedman J. The elements of statistical learning. Elements. 2009;1:337-387. 34. Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY: Association for Computing Machinery; 2016:785-784. 35. Backman R, Bayliss S, Moore D, Litchfield I. Clinical reminder alert fatigue in healthcare: a systematic literature review protocol using qualitative evidence. Syst Rev. 2017;6(1):255. doi:10.1186/s13643-017-0627-z 36. Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R; with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17(1):36. doi:10.1186/s12911-017-0430-8 37. Johnson HA. Diminishing returns on the road to diagnostic certainty. JAMA. 1991;265(17):2229-2231. doi:10. 1001/jama.1991.03460170083038 38. Krasowski MD, Chudzik D, Dolezal A, et al. Promoting improved utilization of laboratory testing through changes in an electronic medical record: experience at an academic medical center. BMC Med Inform Decis Mak. 2015;15(February):11. doi:10.1186/s12911-015-0137-7 39. Chen JH, Fang DZ, Tim Goodnough L, Evans KH, Lee Porter M, Shieh L. Why providers transfuse blood products outside recommended guidelines in spite of integrated electronic best practice alerts. JHospMed. 2015; 10(1):1-7. doi:10.1002/jhm.2236 40. Pageler NM, Franzon D, Longhurst CA, et al. Embedding time-limited laboratory orders within computerized provider order entry reduces laboratory utilization. Pediatr Crit Care Med. 2013;14(4):413-419. doi:10.1097/PCC. 0b013e318272010c 41. Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980;302(20): 1109-1117. doi:10.1056/NEJM198005153022003 42. Kassirer JP, Moskowitz AJ, Lau J, Pauker SG. Decision analysis: a progress report. Ann Intern Med. 1987;106 (2):275-291. doi:10.7326/0003-4819-106-2-275 43. Rosenbloom ST, Chiu K-W, Byrne DW, Talbert DA, Neilson EG, Miller RA. Interventions to regulate ordering of serum magnesium levels: report of an unintended consequence of decision support. J Am Med Inform Assoc. 2005;12(5):546-553. doi:10.1197/jamia.M1811 JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 12/13 JAMA Network Open | Health Informatics Prevalence and Predictability of Low-Yield Inpatient Laboratory Diagnostic Tests 44. Corson AH, Fan VS, White T, et al. A multifaceted hospitalist quality improvement intervention: decreased frequency of common labs. JHospMed. 2015;10(6):390-395. doi:10.1002/jhm.2354 45. Sadowski BW, Lane AB, Wood SM, Robinson SL, Kim CH. High-value, cost-conscious care: iterative systems- based interventions to reduce unnecessary laboratory testing. Am J Med. 2017;130(9):1112.e1-1112.e7. doi:10.1016/ j.amjmed.2017.02.029 46. Afshar N, Tabas J, Afshar K, Silbergleit R. Blood cultures for community-acquired pneumonia: are they worthy of two quality measures? a systematic review. JHospMed. 2009;4(2):112-123. doi:10.1002/jhm.382 47. Vukkadala N, Auerbach A. Clinical utility of routine CBC testing in patients with community-acquired pneumonia. JHospMed. 2017;12(5):336-338. doi:10.12788/jhm.2734 48. Last M. Online classification of nonstationary data streams. Intell Data Anal. 2002;6(2):129-147. doi:10.3233/ IDA-2002-6203 SUPPLEMENT. eFigure 1. Machine Learning Pipeline eFigure 2. ROC Curves for Stanford Standalone Labs eFigure 3. ROC Curves for Stanford Components eFigure 4. ROC Curves for UMich Standalone Labs eFigure 5. ROC Curves for UMich Components eFigure 6. ROC Curves for UCSF Standalone Labs eFigure 7. ROC Curves for UCSF Components eTable 1. Data Matrix Feature Summary eTable 2. Model Construction Summary eTable 3. Diagnostic Metrics for Top Stanford Standalone Labs eTable 4. Diagnostic Metrics for Common Stanford Components eTable 5. Diagnostic Metrics for Top UMich Standalone Labs eTable 6. Diagnostic Metrics for Common UMich Components eTable 7. Diagnostic Metrics for Top UCSF Standalone Labs eTable 8. Diagnostic Metrics for Common UCSF Components eTable 9. Diagnostic Metrics for Common Components in Transferability Study eTable 10. Medicare and Chargemaster Fees for Standalone Labs eTable 11. Top 3 Important Features for Top Stanford Standalone Labs eTable 12. Top 3 Important Features for Common Stanford Components eTable 13. Top 3 Important Features for Top UMich Standalone Labs eTable 14. Top 3 Important Features for Common UMich Components eTable 15. Top 3 Important Features for Top UCSF Standalone Labs eTable 16. Top 3 Important Features for Common UCSF Components eMethods. Technical Details of Machine Learning Algorithm JAMA Network Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 (Reprinted) September 11, 2019 13/13 Supplementary Online Content Xu S, Hom J, Balasubramanian S, et al. Prevalence and predictability of low-yield inpatient laboratory diagnostic tests. JAMA Netw Open. 2019;2(9):e1910967. doi:10.1001/jamanetworkopen.2019.10967 eFigure 1. Machine Learning Pipeline eFigure 2. ROC Curves for Stanford Standalone Labs eFigure 3. ROC Curves for Stanford Components eFigure 4. ROC Curves for UMich Standalone Labs eFigure 5. ROC Curves for UMich Components eFigure 6. ROC Curves for UCSF Standalone Labs eFigure 7. ROC Curves for UCSF Components eTable 1. Data Matrix Feature Summary eTable 2. Model Construction Summary eTable 3. Diagnostic Metrics for Top Stanford Standalone Labs eTable 4. Diagnostic Metrics for Common Stanford Components eTable 5. Diagnostic Metrics for Top UMich Standalone Labs eTable 6. Diagnostic Metrics for Common UMich Components eTable 7. Diagnostic Metrics for Top UCSF Standalone Labs eTable 8. Diagnostic Metrics for Common UCSF Components eTable 9. Diagnostic Metrics for Common Components in Transferability Study eTable 10. Medicare and Chargemaster Fees for Standalone Labs eTable 11. Top 3 Important Features for Top Stanford Standalone Labs eTable 12. Top 3 Important Features for Common Stanford Components eTable 13. Top 3 Important Features for Top UMich Standalone Labs eTable 14. Top 3 Important Features for Common UMich Components eTable 15. Top 3 Important Features for Top UCSF Standalone Labs eTable 16. Top 3 Important Features for Common UCSF Components eMethods. Technical Details of Machine Learning Algorithm This supplementary material has been provided by the authors to give readers additional information about their work. © 2019 Xu S et al. JAMA Network Open. eFigure 1. Machine Learning Pipeline Data processing, machine learning, and statistical analysis pipeline. When applied to a different dataset, an extra data extraction step will need to be implemented that extracts raw data (labs, diagnoses, demographics, encounters, patient info, treatment teams, etc.) from their database into Stanford-like columns The prediction output is a list of predicted normality scores, which were then compared against actual order results (labels). ROC: receiver operating characteristic, AUROC (or C-statistic): Area Under the ROC curve, NPV: negative predictive value, PPV: positive predictive value. © 2019 Xu S et al. JAMA Network Open. © 2019 Xu S et al. JAMA Network Open. eFigure 2. ROC Curves for Stanford Standalone Labs eFigure 3. ROC Curves for Stanford Components © 2019 Xu S et al. JAMA Network Open. eFigure 4. ROC Curves for UMich Standalone Labs eFigure 5. ROC Curves for UMich Components © 2019 Xu S et al. JAMA Network Open. eFigure 6. ROC Curves for UCSF Standalone Labs eFigure 7. ROC Curves for UCSF Components © 2019 Xu S et al. JAMA Network Open. category type features Lab of binary Normalilty of the Most Recent Order Interest time Time of Order (Month), Time of Order (Hour) Features modeled as nominal value, sine, and cosine. time-binned Order History counts Events represent prior orders of lab of interest. Demograph integer Age ics binary Female, Male Asian, Black, Hispanic/Latino, Native American, Pacific Islander, White Hispanic/Latino, White Non- Hispanic/Latino, Other Race, Unknown Race Admission integer Days since Admission Treatment time-binned Cardiology, Cardiovascular ICU, Coronary Care Unit, Team counts Hematology/Oncology, Medical ICU, Medicine, Neurology, Psychiatry, Surgery, Surgical ICU, Transplant, Trauma Events represent patient being treated by specialty. Flow Sheet summary Diastolic Blood Pressure, FiO2, Glasgow Coma Scale statistics Score, Pulse, Respiration, Systolic Blood Pressure, Temperature, Urine Comorbiditi time-binned Cerebrovascular Disease, Chronic Obstructive es counts Pulmonary Disease (COPD), Congestive Heart Failure, Dementia, Diabetes, Diabetes Complications, Hemiplegia/Paraplegia, HIV/AIDS, Liver Damage (Mild), Liver Damage (Moderate to Severe), Malignancy, Metastatic Malignancy, Myocardial Infarction, Peptic Ulcer, Peripheral Vascular Disease, Renal Disease, Rheumatism Events represent comorbidity being added to problem list. Lab Results summary Albumin, Arterial CO2 Partial Pressure, Arterial O2 statistics Partial Pressure, Arterial pH, Blood Urea Nitrogen, C- reactive Protein, Calcium, CO2, Creatinine, Erythrocyte Sedimentation Rate, Hematocrit, Lactate, Platelet Count, Potassium, Sodium, Total Bilirubin, Troponin I, Venous CO2 Partial Pressure, Venous O2 Partial Pressure, Venous pH, White Blood Cell Count eTable 1. Data Matrix Feature Summary For each lab we studied, we constructed an M-by-N data matrix. Each of the N columns represents a single feature to potentially be used in the prediction of normal results for the lab of interest. In total, ~880 features were used for building each model (the precise value varied per lab test, based on the number of component results returned by the © 2019 Xu S et al. JAMA Network Open. clinical laboratory). The table shows the categories of features and their data types. Each of the time-binned counts represents an event aggregated at various time intervals (1, 2, 4, 7, 14, 30, 90, 180, 365, 730, and 1460 days before lab order) and days since last event. Each of the summary statistics represents a value aggregated over the past 3 (for vitals) or 14 days (for lab results) by count, normal count, minimum, maximum, median, standard deviation, first value, last value, slope, and days since first and last values. Algorithm Hyperparameters AdaBoost adaboost_algorithm: SAMME.R, base_estimator: decision-tree, class_weight: balanced, learning_rate: [0.001, 0.01, 0.1, 1.0, 10.0], n_estimators: [10, 20, 30, 40, 50] Decision Tree class_weight: balanced, criterion: gini, max_depth: [1, 2, 3, 4, 5, None], max_features: [sqrt, log2, None], max_leaf_nodes: None, min_impurity_decrease: 0.0, min_samples_leaf: [0.01, 0.1, 1.0, 10.0], min_samples_split: [0.02, 0.2, 2, 20], min_weight_fraction_leaf: 0.0, presort: None, splitter: best Gaussian Naive priors: [[0.0001, 0.9999], [0.001, 0.999], [0.01, 0.99], [0.05, 0.95], Bayes [0.1, 0.9], [0.25, 0.75], [0.5, 0.5], [0.75, 0.25], [0.9, 0.1], [0.95, 0.05], [0.99, 0.01], [0.999, 0.001], [0.9999, 0.0001]] L1 Logistic C: [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0], Regression class_weight: balanced, dual: False, fit_intercept: True, max_iter: 1024, penalty: L1, solver: SAGA, tol: 0.0001 Neural Network Activation: logistic, tanh, relu Layer_sizes: (880, 10, 1), (880, 10, 10, 10, 1), (880, 10, 10, 10, 10, 10, 1) Solver: lbgfs, sgd, adam Random Forest bootstrap: True, n_estimators: [2, 5, 10, 15, 20, 25], warm_start: False © 2019 Xu S et al. JAMA Network Open. Other hyperparameters same as Decision Tree. Regress and M: [1, 2, 3, 4, 5] Round Other hyperparameters same as L1 Logistic Regression. XGBoost Colsample_bytree: [0.6, 0.8, 1.0] Learning_rate: [0.001, 0.01, 0.1, 1.0, 10.0] Max_depth: [1, 2, 3, 4, 5] Min_child_weight: [1, 5, 10] Subsample: [0.6, 0.8, 1.0] eTable 2. Model Construction Summary Eight machine learning algorithms were applied to predict lab result. Each algorithm tune the performance of the algorithms. Hyperparameters with multiple values describe the hyperparameter search space, which was explored exhaustively and scored based on AUROC (C-statistics) through 10-fold cross validation. Lab Test Vol C 95% Prev Target NPV Sens Spec NPV Magnesium 4246 0.76 [0.74, 0.78] 26% 0.99 95% 29% 98% 17% 12% 0.60% 25% 62% Magnesium 4246 0.76 [0.74, 0.78] 26% 0.95 91% 36% 86% 47% 35% 3.6% 22% 39% Magnesium 4246 0.76 [0.74, 0.78] 26% 0.9 87% 43% 71% 67% 50% 7.6% 18% 24% Magnesium 4246 0.76 [0.73, 0.78] 26% 0.8 79% 63% 29% 94% 70% 18% 7.5% 4.4% Prothrombin Time 2244 0.89 [0.88, 0.91] 80% 0.99 92% 81% 100% 2.3% 0.45% 0.04% 80% 19% Prothrombin Time 2244 0.89 [0.88, 0.91] 80% 0.95 85% 81% 100% 3.6% 0.70% 0.12% 80% 19% Prothrombin Time 2244 0.89 [0.88, 0.91] 80% 0.9 82% 83% 99% 18% 3.4% 0.74% 80% 16% Prothrombin Time 2244 0.89 [0.88, 0.90] 80% 0.8 69% 87% 96% 39% 7.6% 3.4% 77% 12% Phosphorus 2120 0.74 [0.72, 0.76] 33% 0.9 84% 45% 78% 54% 36% 7.1% 26% 31% Phosphorus 2120 0.74 [0.72, 0.76] 33% 0.95 88% 39% 91% 30% 20% 2.8% 30% 47% Phosphorus 2120 0.74 [0.72, 0.76] 33% 0.8 77% 57% 52% 81% 55% 16% 17% 13% Phosphorus 2120 0.74 [0.72, 0.76] 33% 0.99 88% 34% 98% 6.7% 4.5% 0.63% 32% 63% Partial Thromboplastin 1471 0.86 [0.85, 0.87] 61% 0.99 90% 62% 100% 4.1% 1.6% 0.18% 61% 37% Time Partial Thromboplastin 1471 0.86 [0.84, 0.87] 61% 0.9 81% 69% 96% 30% 12% 2.8% 59% 27% Time Partial Thromboplastin 1471 0.86 [0.85, 0.87] 61% 0.8 74% 78% 87% 61% 23% 8.2% 53% 15% Time Partial Thromboplastin 1471 0.86 [0.85, 0.87] 61% 0.95 87% 65% 98% 17% 6.5% 1.0% 60% 32% Time Lactate 1230 0.87 [0.85, 0.88] 29% 0.8 77% 90% 28% 99% 71% 21% 8.0% 0.86% © 2019 Xu S et al. JAMA Network Open. Lactate 1230 0.87 [0.85, 0.88] 29% 0.95 91% 56% 82% 74% 53% 5.2% 23% 19% Lactate 1230 0.87 [0.85, 0.89] 29% 0.99 97% 35% 98% 28% 20% 0.54% 28% 51% Lactate 1230 0.87 [0.85, 0.89] 29% 0.9 87% 72% 65% 90% 64% 9.9% 19% 7.4% Calcium Ionized 1197 0.72 [0.70, 0.74] 61% 0.95 90% 62% 100% 4.8% 1.9% 0.21% 61% 37% Calcium Ionized 1197 0.72 [0.69, 0.74] 61% 0.9 82% 63% 99% 9.0% 3.5% 0.80% 60% 36% Calcium Ionized 1197 0.72 [0.70, 0.74] 61% 0.8 71% 66% 93% 25% 10% 4.1% 57% 29% Calcium Ionized 1197 0.72 [0.70, 0.74] 61% 0.99 96% 61% 100% 2.4% 0.93% 0.04% 61% 38% Potassium 752 0.81 [0.79, 0.84] 12% 0.95 92% 40% 43% 91% 80% 7.0% 5.2% 7.9% Potassium 752 0.81 [0.79, 0.84] 12% 0.8 88% - 0 100% 88% 12% 0 0 Potassium 752 0.81 [0.79, 0.84] 12% 0.9 88% 64% 2.8% 100% 88% 12% 0.34% 0.19% Potassium 752 0.81 [0.79, 0.84] 12% 0.99 97% 21% 89% 54% 48% 1.3% 11% 40% Troponin I 534 0.92 [0.91, 0.93] 33% 0.99 95% 38% 98% 23% 16% 0.79% 32% 52% Troponin I 534 0.92 [0.91, 0.93] 33% 0.95 93% 67% 88% 79% 53% 4.0% 29% 14% Troponin I 534 0.92 [0.91, 0.93] 33% 0.9 89% 88% 76% 95% 64% 7.8% 25% 3.3% Troponin I 534 0.92 [0.91, 0.93] 33% 0.8 79% 99% 45% 100% 67% 18% 15% 0.16% LDH Total 455 0.93 [0.93, 0.94] 47% 0.8 79% 90% 72% 93% 50% 13% 34% 3.8% LDH Total 455 0.93 [0.93, 0.94] 47% 0.9 90% 78% 90% 78% 42% 4.6% 42% 12% LDH Total 455 0.93 [0.93, 0.94] 47% 0.99 98% 60% 99% 43% 23% 0.35% 46% 30% LDH Total 455 0.93 [0.93, 0.94] 47% 0.95 95% 71% 96% 65% 35% 1.8% 45% 18% Heparin 423 0.76 [0.74, 0.78] 63% 0.99 0 63% 100% 0 0 0.04% 63% 37% Heparin 423 0.76 [0.74, 0.78] 63% 0.8 72% 71% 92% 37% 14% 5.2% 58% 24% Heparin 423 0.76 [0.74, 0.78] 63% 0.9 74% 68% 95% 24% 8.9% 3.2% 60% 28% Heparin 423 0.76 [0.74, 0.78] 63% 0.95 0 63% 100% 0 0 0.04% 63% 37% Urinalysis 417 0.71 [0.63, 0.80] 80% 0.95 - 80% 100% 0 0 0 80% 20% Urinalysis 417 0.71 [0.63, 0.79] 80% 0.8 - 80% 100% 0 0 0 80% 20% Urinalysis 417 0.71 [0.63, 0.79] 80% 0.99 - 80% 100% 0 0 0 80% 20% Urinalysis 417 0.71 [0.63, 0.79] 80% 0.9 - 80% 100% 0 0 0 80% 20% Blood Culture (Aerobic & 400 0.66 [0.61, 0.71] 8.1% 0.8 92% - 0 100% 92% 8.1% 0 0 Anaerobic) Blood Culture (Aerobic & 400 0.66 [0.61, 0.71] 8.1% 0.9 92% - 0 100% 92% 8.1% 0 0 Anaerobic) Blood Culture (Aerobic & 400 0.66 [0.61, 0.71] 8.1% 0.95 93% 16% 16% 93% 85% 6.8% 1.3% 6.6% Anaerobic) Blood Culture (Aerobic & 400 0.66 [0.61, 0.71] 8.1% 0.99 94% 14% 45% 76% 70% 4.5% 3.6% 22% Anaerobic) Blood Culture (2 371 0.62 [0.58, 0.67] 9.1% 0.95 93% 12% 61% 54% 49% 3.6% 5.6% 42% Aerobic) Blood Culture (2 371 0.62 [0.58, 0.67] 9.1% 0.8 91% - 0 100% 91% 9.1% 0 0 Aerobic) Blood Culture (2 371 0.62 [0.58, 0.67] 9.1% 0.9 91% - 0 100% 91% 9.1% 0 0 Aerobic) © 2019 Xu S et al. JAMA Network Open. Blood Culture (2 371 0.62 [0.57, 0.67] 9.1% 0.99 - 9.1% 100% 0 0 0 9.1% 91% Aerobic) Sodium 361 0.92 [0.91, 0.93] 57% 0.99 100% 58% 100% 3.1% 1.4% 0 57% 42% Sodium 361 0.92 [0.91, 0.93] 57% 0.95 93% 66% 98% 35% 15% 1.1% 56% 28% Sodium 361 0.92 [0.91, 0.93] 57% 0.8 76% 91% 78% 90% 39% 12% 45% 4.5% Sodium 361 0.92 [0.91, 0.93] 57% 0.9 87% 79% 92% 68% 29% 4.5% 52% 14% Lidocaine 315 0.83 [0.79, 0.86] 23% 0.99 94% 39% 89% 57% 44% 2.6% 21% 33% Lidocaine 315 0.83 [0.79, 0.86] 23% 0.95 91% 55% 73% 82% 63% 6.3% 17% 14% Lidocaine 315 0.83 [0.79, 0.86] 23% 0.8 77% 100% 0.62% 100% 77% 23% 0.14% 0 Lidocaine 315 0.83 [0.79, 0.86] 23% 0.9 83% 61% 35% 93% 71% 15% 8.2% 5.3% Hematocrit 288 0.9 [0.88, 0.92] 93% 0.99 100% 93% 100% 1.0% 0.08% 0 93% 7.2% Hematocrit 288 0.9 [0.87, 0.92] 93% 0.8 50% 94% 99% 16% 1.1% 1.1% 92% 6.1% Hematocrit 288 0.9 [0.88, 0.92] 93% 0.9 44% 93% 99% 7.9% 0.57% 0.72% 92% 6.7% Hematocrit 288 0.9 [0.88, 0.92] 93% 0.95 58% 93% 100% 3.7% 0.27% 0.19% 93% 7.0% Urine Culture 257 0.71 [0.68, 0.74] 36% 0.99 100% 36% 100% 2.2% 1.4% 0 36% 63% Urine Culture 257 0.71 [0.68, 0.74] 36% 0.95 90% 41% 95% 24% 15% 1.7% 34% 49% Urine Culture 257 0.71 [0.68, 0.74] 36% 0.9 84% 44% 86% 40% 26% 5.0% 31% 39% Urine Culture 257 0.71 [0.68, 0.74] 36% 0.8 75% 52% 58% 71% 45% 15% 21% 19% Urinalysis With 246 0.63 [0.57, 0.70] 72% 0.9 40% 75% 86% 24% 6.6% 10% 62% 21% Microscopic Urinalysis With 246 0.63 [0.56, 0.70] 72% 0.8 35% 75% 78% 32% 8.8% 16% 56% 19% Microscopic Urinalysis With 246 0.63 [0.57, 0.70] 72% 0.95 48% 75% 93% 17% 4.7% 5.0% 67% 23% Microscopic Urinalysis With 246 0.63 [0.57, 0.70] 72% 0.99 43% 73% 97% 6.8% 1.9% 2.5% 70% 26% Microscopic Uric Acid 229 0.97 [0.96, 0.98] 3.8% 0.99 98% 51% 50% 98% 94% 1.9% 1.9% 1.8% Uric Acid 229 0.97 [0.96, 0.98] 3.8% 0.95 96% - 0 100% 96% 3.8% 0 0 Uric Acid 229 0.97 [0.96, 0.98] 3.8% 0.8 96% - 0 100% 96% 3.8% 0 0 Uric Acid 229 0.97 [0.96, 0.98] 3.8% 0.9 96% - 0 100% 96% 3.8% 0 0 Hemoglobin A1c 225 0.81 [0.79, 0.82] 59% 0.95 87% 63% 99% 14% 5.8% 0.86% 58% 35% Hemoglobin A1c 225 0.81 [0.79, 0.82] 59% 0.99 93% 61% 100% 6.8% 2.8% 0.20% 59% 38% Hemoglobin A1c 225 0.81 [0.79, 0.82] 59% 0.9 85% 66% 97% 28% 11% 2.0% 57% 29% Hemoglobin A1c 225 0.81 [0.79, 0.82] 59% 0.8 73% 74% 86% 55% 23% 8.4% 51% 18% Sepsis Protocol Lactate 187 0.86 [0.84, 0.88] 17% 0.99 97% 28% 92% 52% 44% 1.3% 15% 40% Sepsis Protocol Lactate 187 0.86 [0.84, 0.88] 17% 0.8 83% - 0 100% 83% 17% 0 0 Sepsis Protocol Lactate 187 0.86 [0.84, 0.88] 17% 0.9 87% 80% 30% 98% 82% 12% 5.0% 1.3% Sepsis Protocol Lactate 187 0.86 [0.84, 0.88] 17% 0.95 92% 55% 60% 90% 75% 6.7% 10% 8.4% iSTAT Troponin I 184 0.79 [0.75, 0.82] 13% 0.99 95% 16% 91% 26% 23% 1.1% 12% 64% iSTAT Troponin I 184 0.79 [0.75, 0.82] 13% 0.95 93% 55% 54% 93% 81% 6.2% 7.1% 5.9% iSTAT Troponin I 184 0.79 [0.75, 0.82] 13% 0.9 87% 93% 4.6% 100% 87% 13% 0.62% 0.05% © 2019 Xu S et al. JAMA Network Open. iSTAT Troponin I 184 0.79 [0.75, 0.82] 13% 0.8 87% - 0 100% 87% 13% 0 0 Platelet Count 174 0.95 [0.94, 0.96] 44% 0.99 98% 57% 99% 41% 23% 0.44% 43% 33% Platelet Count 174 0.95 [0.94, 0.96] 44% 0.95 92% 76% 92% 78% 44% 3.6% 40% 12% Platelet Count 174 0.95 [0.94, 0.96] 44% 0.9 89% 90% 85% 92% 52% 6.6% 37% 4.2% Platelet Count 174 0.95 [0.94, 0.96] 44% 0.8 80% 98% 68% 99% 55% 14% 30% 0.67% Lipase 174 0.79 [0.77, 0.82] 23% 0.99 93% 26% 96% 17% 13% 0.98% 22% 64% Lipase 174 0.79 [0.77, 0.81] 23% 0.95 91% 35% 82% 53% 41% 4.3% 19% 36% Lipase 174 0.79 [0.77, 0.82] 23% 0.8 80% 92% 15% 100% 76% 20% 3.6% 0.30% Lipase 174 0.79 [0.77, 0.82] 23% 0.9 88% 55% 61% 85% 65% 9.1% 14% 11% Procalcitonin 174 0.89 [0.88, 0.90] 52% 0.95 91% 61% 97% 34% 17% 1.6% 50% 32% Procalcitonin 174 0.89 [0.88, 0.90] 52% 0.9 85% 73% 89% 65% 31% 5.5% 46% 17% Procalcitonin 174 0.89 [0.88, 0.90] 52% 0.8 77% 86% 75% 87% 42% 13% 39% 6.2% Procalcitonin 174 0.89 [0.88, 0.90] 52% 0.99 96% 53% 100% 5.7% 2.8% 0.12% 52% 46% Lactic Acid 152 0.87 [0.85, 0.88] 25% 0.9 87% 72% 59% 92% 69% 10% 15% 5.8% Lactic Acid 152 0.87 [0.85, 0.88] 25% 0.8 79% 95% 20% 100% 75% 20% 5.0% 0.25% Lactic Acid 152 0.87 [0.85, 0.88] 25% 0.95 92% 54% 79% 78% 58% 5.3% 20% 17% Lactic Acid 152 0.87 [0.85, 0.88] 25% 0.99 97% 34% 97% 36% 27% 0.85% 24% 48% Fibrinogen 148 0.78 [0.76, 0.80] 61% 0.99 100% 61% 100% 0.41% 0.16% 0 61% 39% Fibrinogen 148 0.78 [0.76, 0.80] 61% 0.95 79% 63% 98% 11% 4.1% 1.1% 60% 35% Fibrinogen 148 0.78 [0.77, 0.80] 61% 0.9 76% 67% 95% 27% 10% 3.3% 58% 29% Fibrinogen 148 0.78 [0.76, 0.80] 61% 0.8 74% 72% 90% 44% 17% 6.1% 55% 22% Thyroid Stimulating 145 0.64 [0.62, 0.67] 27% 0.9 81% 34% 67% 52% 38% 9.0% 18% 35% Hormone Thyroid Stimulating 145 0.64 [0.62, 0.67] 27% 0.8 77% 54% 26% 92% 67% 20% 7.1% 6.0% Hormone Thyroid Stimulating 145 0.64 [0.62, 0.67] 27% 0.99 84% 27% 100% 0.87% 0.63% 0.12% 27% 72% Hormone Thyroid Stimulating 145 0.64 [0.62, 0.67] 27% 0.95 83% 30% 87% 24% 18% 3.7% 24% 55% Hormone Creatine Kinase 126 0.94 [0.93, 0.95] 48% 0.9 87% 88% 86% 89% 47% 6.8% 41% 5.6% Creatine Kinase 126 0.94 [0.93, 0.95] 48% 0.8 80% 96% 73% 97% 51% 13% 35% 1.4% Creatine Kinase 126 0.94 [0.93, 0.95] 48% 0.99 95% 66% 97% 55% 29% 1.5% 46% 24% Creatine Kinase 126 0.94 [0.93, 0.95] 48% 0.95 91% 79% 92% 78% 41% 4.0% 44% 12% C-Reactive Protein 125 0.86 [0.84, 0.88] 87% 0.99 82% 87% 100% 3.9% 0.52% 0.11% 87% 13% C-Reactive Protein 125 0.86 [0.84, 0.88] 87% 0.8 68% 89% 98% 22% 3.0% 1.4% 85% 10% C-Reactive Protein 125 0.86 [0.84, 0.88] 87% 0.95 81% 87% 100% 6.1% 0.81% 0.18% 86% 13% C-Reactive Protein 125 0.86 [0.84, 0.88] 87% 0.9 72% 88% 99% 12% 1.6% 0.59% 86% 12% -proBNP 122 0.81 [0.79, 0.83] 82% 0.8 74% 84% 99% 17% 3.2% 1.1% 81% 15% -proBNP 122 0.81 [0.79, 0.83] 82% 0.9 76% 83% 99% 9.4% 1.7% 0.53% 81% 17% -proBNP 122 0.81 [0.79, 0.83] 82% 0.95 70% 82% 100% 1.6% 0.29% 0.12% 82% 18% -proBNP 122 0.81 [0.79, 0.83] 82% 0.99 71% 82% 100% 1.1% 0.20% 0.08% 82% 18% © 2019 Xu S et al. JAMA Network Open. Triglycerides 105 0.86 [0.84, 0.87] 42% 0.8 76% 84% 61% 91% 53% 17% 26% 5.0% Triglycerides 105 0.86 [0.84, 0.87] 42% 0.9 84% 66% 82% 69% 40% 7.5% 35% 18% Triglycerides 105 0.86 [0.84, 0.87] 42% 0.95 92% 54% 95% 41% 24% 2.1% 40% 34% Triglycerides 105 0.86 [0.84, 0.87] 42% 0.99 95% 47% 99% 19% 11% 0.55% 42% 46% CK, MB 96 0.9 [0.88, 0.91] 49% 0.8 79% 86% 75% 88% 45% 12% 37% 5.9% CK, MB 96 0.9 [0.88, 0.91] 49% 0.9 86% 74% 88% 70% 36% 5.7% 43% 15% CK, MB 96 0.9 [0.88, 0.91] 49% 0.95 91% 64% 95% 48% 24% 2.4% 46% 27% CK, MB 96 0.9 [0.88, 0.91] 49% 0.99 96% 53% 99% 15% 7.7% 0.35% 49% 43% C.diff Toxin B Gene 95 0.65 [0.62, 0.68] 18% 0.8 82% - 0 100% 82% 18% 0 0 C.diff Toxin B Gene 95 0.65 [0.62, 0.68] 18% 0.9 87% 27% 55% 67% 55% 8.2% 9.9% 27% C.diff Toxin B Gene 95 0.65 [0.62, 0.68] 18% 0.95 89% 21% 85% 28% 23% 2.7% 15% 59% C.diff Toxin B Gene 95 0.65 [0.62, 0.68] 18% 0.99 80% 18% 99% 1.2% 0.96% 0.24% 18% 81% Osmolality 88 0.92 [0.91, 0.93] 53% 0.8 76% 91% 75% 91% 43% 13% 40% 4.0% Osmolality 88 0.92 [0.91, 0.93] 53% 0.9 88% 82% 91% 77% 36% 4.9% 48% 11% Osmolality 88 0.92 [0.91, 0.93] 53% 0.95 94% 70% 97% 53% 25% 1.7% 52% 22% Osmolality 88 0.92 [0.91, 0.93] 53% 0.99 96% 56% 100% 11% 5.4% 0.22% 53% 41% Respiratory Culture And 87 0.62 [0.58, 0.65] 35% 0.99 80% 35% 100% 0.59% 0.38% 0.10% 34% 65% Gram Stain Respiratory Culture And 87 0.62 [0.58, 0.65] 35% 0.95 92% 35% 100% 1.8% 1.2% 0.10% 34% 64% Gram Stain Respiratory Culture And 87 0.62 [0.58, 0.65] 35% 0.9 78% 36% 92% 15% 9.5% 2.7% 32% 56% Gram Stain Respiratory Culture And 87 0.62 [0.58, 0.65] 35% 0.8 72% 42% 55% 61% 40% 15% 19% 26% Gram Stain Sedimentation Rate 85 0.83 [0.81, 0.85] 69% 0.99 100% 70% 100% 2.0% 0.62% 0 69% 30% (ESR) Sedimentation Rate 85 0.83 [0.81, 0.84] 69% 0.95 86% 73% 99% 19% 5.9% 1.0% 68% 25% (ESR) Sedimentation Rate 85 0.83 [0.81, 0.84] 69% 0.9 80% 75% 97% 30% 9.1% 2.3% 67% 22% (ESR) Sedimentation Rate 85 0.83 [0.81, 0.85] 69% 0.8 70% 78% 92% 41% 13% 5.4% 64% 18% (ESR) Ferritin 82 0.81 [0.79, 0.82] 62% 0.95 91% 63% 100% 4.4% 1.7% 0.16% 62% 36% Ferritin 82 0.81 [0.79, 0.82] 62% 0.8 73% 72% 91% 42% 16% 5.8% 57% 22% Ferritin 82 0.81 [0.79, 0.82] 62% 0.9 80% 66% 97% 17% 6.6% 1.6% 61% 31% Ferritin 82 0.81 [0.79, 0.82] 62% 0.99 93% 63% 100% 2.6% 0.98% 0.08% 62% 37% Albumin 78 0.9 [0.88, 0.91] 89% 0.95 75% 90% 99% 15% 1.7% 0.57% 88% 9.7% Albumin 78 0.9 [0.88, 0.91] 89% 0.9 71% 91% 99% 26% 2.9% 1.2% 87% 8.5% Albumin 78 0.9 [0.88, 0.91] 89% 0.8 60% 92% 97% 35% 4.0% 2.6% 86% 7.4% Albumin 78 0.9 [0.87, 0.91] 89% 0.99 74% 89% 100% 5.0% 0.57% 0.20% 88% 11% Ammonia 77 0.78 [0.76, 0.79] 59% 0.8 73% 69% 90% 39% 16% 5.9% 54% 25% Ammonia 77 0.78 [0.76, 0.79] 59% 0.9 78% 63% 97% 17% 6.7% 1.9% 58% 34% © 2019 Xu S et al. JAMA Network Open. Ammonia 77 0.78 [0.76, 0.80] 59% 0.95 80% 61% 99% 8.3% 3.4% 0.85% 59% 37% Ammonia 77 0.78 [0.76, 0.79] 59% 0.99 77% 60% 99% 3.4% 1.4% 0.42% 59% 39% Specific Gravity 76 0.67 [0.54, 0.79] 0.85% 0.99 99% - 0 100% 99% 0.85% 0 0 Specific Gravity 76 0.67 [0.55, 0.79] 0.85% 0.95 99% - 0 100% 99% 0.85% 0 0 Specific Gravity 76 0.67 [0.55, 0.78] 0.85% 0.9 99% - 0 100% 99% 0.85% 0 0 Specific Gravity 76 0.67 [0.54, 0.79] 0.85% 0.8 99% - 0 100% 99% 0.85% 0 0 Fungal Culture 74 0.76 [0.73, 0.79] 18% 0.8 82% - 0 100% 82% 18% 0 0 Fungal Culture 74 0.76 [0.73, 0.80] 18% 0.9 93% 28% 81% 54% 45% 3.5% 15% 37% Fungal Culture 74 0.76 [0.72, 0.80] 18% 0.95 - 18% 100% 0 0 0 18% 82% Fungal Culture 74 0.76 [0.73, 0.80] 18% 0.99 - 18% 100% 0 0 0 18% 82% Haptoglobin 71 0.77 [0.75, 0.79] 43% 0.8 74% 65% 66% 73% 42% 14% 28% 15% Haptoglobin 71 0.77 [0.75, 0.79] 43% 0.9 83% 53% 89% 40% 23% 4.8% 38% 34% Haptoglobin 71 0.77 [0.75, 0.79] 43% 0.95 88% 47% 96% 20% 11% 1.6% 41% 46% Haptoglobin 71 0.77 [0.75, 0.79] 43% 0.99 97% 44% 100% 5.5% 3.2% 0.09% 43% 54% Anaerobic Culture 69 0.78 [0.75, 0.82] 13% 0.9 90% 51% 25% 97% 84% 9.5% 3.2% 3.0% Anaerobic Culture 69 0.78 [0.75, 0.82] 13% 0.99 99% 17% 97% 33% 29% 0.41% 12% 58% Anaerobic Culture 69 0.78 [0.75, 0.82] 13% 0.95 94% 28% 64% 76% 66% 4.6% 8.1% 21% Anaerobic Culture 69 0.78 [0.75, 0.82] 13% 0.8 87% - 0 100% 87% 13% 0 0 Fluid Culture And Gram 69 0.73 [0.69, 0.76] 26% 0.99 91% 29% 94% 20% 15% 1.5% 25% 59% Stain Fluid Culture And Gram 69 0.73 [0.69, 0.76] 26% 0.95 85% 41% 67% 66% 49% 8.7% 17% 25% Stain Fluid Culture And Gram 69 0.73 [0.69, 0.76] 26% 0.9 80% 50% 42% 85% 63% 15% 11% 11% Stain Fluid Culture And Gram 69 0.73 [0.69, 0.77] 26% 0.8 76% 67% 10% 98% 73% 23% 2.7% 1.4% Stain Cmv Dna Pcr Quant 66 0.79 [0.76, 0.82] 27% 0.8 79% 90% 28% 99% 72% 20% 7.7% 0.85% Cmv Dna Pcr Quant 66 0.79 [0.76, 0.82] 27% 0.9 85% 61% 58% 86% 63% 11% 16% 10% Cmv Dna Pcr Quant 66 0.79 [0.76, 0.82] 27% 0.95 88% 44% 76% 63% 46% 6.5% 21% 27% Cmv Dna Pcr Quant 66 0.79 [0.76, 0.82] 27% 0.99 92% 31% 95% 21% 15% 1.3% 26% 57% Blood Cult Central Line 62 0.65 [0.60, 0.69] 11% 0.95 90% 20% 19% 90% 80% 8.9% 2.1% 8.5% Catheter By Nurse Blood Cult Central Line 62 0.65 [0.60, 0.70] 11% 0.8 89% - 0 100% 89% 11% 0 0 Catheter By Nurse Blood Cult Central Line 62 0.65 [0.60, 0.70] 11% 0.9 89% 100% 2.1% 100% 89% 11% 0.23% 0 Catheter By Nurse Blood Cult Central Line 62 0.65 [0.61, 0.69] 11% 0.99 93% 15% 68% 52% 46% 3.6% 7.4% 43% Catheter By Nurse T4, FREE 61 0.68 [0.64, 0.71] 11% 0.9 89% 100% 0.34% 100% 89% 11% 0.04% 0 T4, FREE 61 0.68 [0.64, 0.71] 11% 0.95 92% 28% 37% 88% 78% 7.1% 4.2% 11% T4, FREE 61 0.68 [0.64, 0.71] 11% 0.99 95% 13% 90% 24% 21% 1.1% 10% 68% T4, FREE 61 0.68 [0.64, 0.71] 11% 0.8 89% - 0 100% 89% 11% 0 0 © 2019 Xu S et al. JAMA Network Open. Transferrin Saturation 60 0.65 [0.60, 0.70] 88% 0.8 - 88% 100% 0 0 0 88% 12% Transferrin Saturation 60 0.65 [0.60, 0.70] 88% 0.99 - 88% 100% 0 0 0 88% 12% Transferrin Saturation 60 0.65 [0.60, 0.70] 88% 0.9 - 88% 100% 0 0 0 88% 12% Transferrin Saturation 60 0.65 [0.60, 0.70] 88% 0.95 - 88% 100% 0 0 0 88% 12% Vitamin B12 60 0.72 [0.70, 0.74] 32% 0.8 77% 54% 50% 80% 54% 16% 16% 14% Vitamin B12 60 0.72 [0.70, 0.74] 32% 0.9 85% 40% 86% 39% 26% 4.5% 28% 42% Vitamin B12 60 0.72 [0.70, 0.74] 32% 0.95 87% 35% 96% 13% 8.8% 1.3% 31% 59% Vitamin B12 60 0.72 [0.70, 0.74] 32% 0.99 93% 32% 100% 0.82% 0.56% 0.04% 32% 67% Blood Cult - First Set 60 0.67 [0.62, 0.71] 11% 0.99 92% 18% 54% 70% 62% 5.0% 6.0% 27% Blood Cult - First Set 60 0.67 [0.62, 0.71] 11% 0.9 89% 0 0 100% 89% 11% 0 0.08% Blood Cult - First Set 60 0.67 [0.62, 0.72] 11% 0.8 89% - 0 100% 89% 11% 0 0 Blood Cult - First Set 60 0.67 [0.62, 0.71] 11% 0.95 90% 25% 16% 94% 84% 9.3% 1.8% 5.2% Prealbumin 59 0.85 [0.83, 0.86] 77% 0.99 77% 77% 100% 1.9% 0.45% 0.14% 77% 23% Prealbumin 59 0.85 [0.83, 0.86] 77% 0.95 90% 79% 100% 13% 3.0% 0.32% 76% 20% Prealbumin 59 0.85 [0.83, 0.86] 77% 0.9 85% 81% 99% 23% 5.4% 0.95% 76% 18% Prealbumin 59 0.85 [0.83, 0.86] 77% 0.8 79% 84% 97% 39% 9.0% 2.4% 74% 14% Osmolality, Urine 58 0.92 [0.90, 0.93] 7.9% 0.95 94% 50% 22% 98% 90% 6.1% 1.8% 1.8% Osmolality, Urine 58 0.92 [0.90, 0.93] 7.9% 0.99 97% 34% 70% 88% 81% 2.4% 5.5% 11% Osmolality, Urine 58 0.92 [0.90, 0.93] 7.9% 0.9 92% - 0 100% 92% 7.9% 0 0 Osmolality, Urine 58 0.92 [0.90, 0.93] 7.9% 0.8 92% - 0 100% 92% 7.9% 0 0 Digoxin 55 0.8 [0.77, 0.83] 22% 0.99 - 22% 100% 0 0 0 22% 78% Digoxin 55 0.8 [0.77, 0.83] 22% 0.95 95% 36% 89% 56% 44% 2.4% 19% 35% Digoxin 55 0.8 [0.77, 0.83] 22% 0.9 92% 44% 78% 72% 57% 4.7% 17% 22% Digoxin 55 0.8 [0.77, 0.83] 22% 0.8 81% 56% 19% 96% 75% 17% 4.1% 3.2% Reticulocyte Count 50 0.79 [0.77, 0.81] 63% 0.9 81% 70% 96% 28% 10% 2.5% 61% 27% Automated Reticulocyte Count 50 0.79 [0.77, 0.81] 63% 0.99 81% 64% 100% 3.3% 1.2% 0.29% 63% 36% Automated Reticulocyte Count 50 0.79 [0.77, 0.81] 63% 0.8 71% 73% 89% 45% 16% 6.9% 56% 20% Automated Reticulocyte Count 50 0.79 [0.77, 0.81] 63% 0.95 85% 66% 99% 11% 4.2% 0.74% 62% 33% Automated Cortisol 48 0.71 [0.68, 0.73] 24% 0.8 77% 78% 6.5% 99% 75% 23% 1.6% 0.45% Cortisol 48 0.71 [0.69, 0.73] 24% 0.9 83% 38% 52% 73% 56% 12% 12% 20% Cortisol 48 0.71 [0.69, 0.73] 24% 0.95 88% 33% 78% 50% 38% 5.3% 19% 38% Cortisol 48 0.71 [0.69, 0.73] 24% 0.99 96% 29% 97% 23% 18% 0.76% 23% 58% Hepatitis B Antigen 45 0.8 [0.73, 0.86] 2.9% 0.95 97% - 0 100% 97% 2.9% 0 0 Hepatitis B Antigen 45 0.8 [0.73, 0.86] 2.9% 0.9 97% - 0 100% 97% 2.9% 0 0 Hepatitis B Antigen 45 0.8 [0.73, 0.86] 2.9% 0.99 98% 60% 22% 100% 97% 2.3% 0.65% 0.43% Hepatitis B Antigen 45 0.8 [0.73, 0.86] 2.9% 0.8 97% - 0 100% 97% 2.9% 0 0 Iron 40 0.74 [0.71, 0.77] 40% 0.99 91% 44% 98% 17% 10% 0.95% 39% 50% © 2019 Xu S et al. JAMA Network Open. Iron 40 0.74 [0.71, 0.78] 40% 0.8 73% 64% 57% 78% 47% 17% 23% 13% Iron 40 0.74 [0.71, 0.77] 40% 0.9 78% 53% 77% 53% 31% 9.1% 31% 28% Iron 40 0.74 [0.71, 0.78] 40% 0.95 81% 48% 88% 36% 22% 5.0% 35% 38% AFB Culture 40 0.68 [0.59, 0.76] 4.9% 0.99 95% 0 0 100% 95% 4.9% 0 0.10% AFB Culture 40 0.68 [0.59, 0.76] 4.9% 0.8 95% - 0 100% 95% 4.9% 0 0 AFB Culture 40 0.68 [0.59, 0.77] 4.9% 0.95 95% 0 0 100% 95% 4.9% 0 0.10% AFB Culture 40 0.68 [0.60, 0.76] 4.9% 0.9 95% - 0 100% 95% 4.9% 0 0 Calcium 39 0.86 [0.84, 0.87] 66% 0.99 89% 67% 100% 4.1% 1.4% 0.17% 66% 32% Calcium 39 0.86 [0.84, 0.87] 66% 0.9 82% 75% 96% 36% 12% 2.7% 64% 21% Calcium 39 0.86 [0.84, 0.87] 66% 0.8 75% 80% 91% 55% 18% 6.3% 60% 15% Calcium 39 0.86 [0.84, 0.87] 66% 0.95 86% 70% 99% 16% 5.5% 0.93% 65% 28% Biopsy/tissue With Gram 36 0.75 [0.72, 0.79] 37% 0.8 72% 62% 42% 85% 54% 21% 16% 9.5% Stain Biopsy/tissue With Gram 36 0.75 [0.72, 0.79] 37% 0.9 82% 54% 76% 63% 40% 8.7% 28% 24% Stain Biopsy/tissue With Gram 36 0.75 [0.72, 0.79] 37% 0.99 88% 38% 99% 5.9% 3.8% 0.54% 36% 60% Stain Biopsy/tissue With Gram 36 0.75 [0.71, 0.79] 37% 0.95 89% 44% 93% 32% 20% 2.4% 34% 43% Stain Afb Culture 35 0.81 [0.68, 0.92] 1.5% 0.8 99% - 0 100% 99% 1.5% 0 0 Afb Culture 35 0.81 [0.66, 0.92] 1.5% 0.9 99% - 0 100% 99% 1.5% 0 0 Afb Culture 35 0.81 [0.66, 0.92] 1.5% 0.95 99% - 0 100% 99% 1.5% 0 0 Afb Culture 35 0.81 [0.65, 0.92] 1.5% 0.99 100% 2.4% 92% 45% 44% 0.12% 1.3% 54% Pregnancy Test 34 0.64 [0.54, 0.74] 1.8% 0.95 98% - 0 100% 98% 1.8% 0 0 Pregnancy Test 34 0.64 [0.54, 0.73] 1.8% 0.9 98% - 0 100% 98% 1.8% 0 0 Pregnancy Test 34 0.64 [0.54, 0.74] 1.8% 0.8 98% - 0 100% 98% 1.8% 0 0 Pregnancy Test 34 0.64 [0.54, 0.73] 1.8% 0.99 - 1.8% 100% 0 0 0 1.8% 98% Urea Nitrogen 34 0.91 [0.89, 0.92] 51% 0.8 77% 89% 74% 90% 45% 13% 38% 4.7% Urea Nitrogen 34 0.91 [0.89, 0.92] 51% 0.9 87% 82% 88% 81% 40% 6.2% 44% 9.5% Urea Nitrogen 34 0.91 [0.89, 0.92] 51% 0.95 90% 75% 93% 68% 33% 3.7% 47% 16% Urea Nitrogen 34 0.91 [0.89, 0.92] 51% 0.99 96% 54% 99% 13% 6.3% 0.27% 50% 43% Gram Stain 31 0.57 [0.52, 0.63] 11% 0.9 - 11% 100% 0 0 0 11% 89% Gram Stain 31 0.57 [0.52, 0.63] 11% 0.99 - 11% 100% 0 0 0 11% 89% Gram Stain 31 0.57 [0.52, 0.63] 11% 0.95 - 11% 100% 0 0 0 11% 89% Gram Stain 31 0.57 [0.52, 0.63] 11% 0.8 - 11% 100% 0 0 0 11% 89% 29 0.66 [0.61, 0.69] 54% 0.8 64% 54% 99% 3.1% 1.4% 0.80% 53% 45% 29 0.66 [0.61, 0.70] 54% 0.9 78% 54% 99% 2.4% 1.1% 0.32% 54% 45% 29 0.66 [0.61, 0.70] 54% 0.95 80% 54% 100% 1.4% 0.64% 0.16% 54% 45% 29 0.66 [0.61, 0.70] 54% 0.99 80% 54% 100% 1.4% 0.64% 0.16% 54% 45% Protein Total 27 0.76 [0.73, 0.78] 67% 0.99 81% 67% 100% 2.6% 0.88% 0.20% 66% 33% Protein Total 27 0.76 [0.73, 0.78] 67% 0.95 83% 68% 99% 8.1% 2.7% 0.54% 66% 31% © 2019 Xu S et al. JAMA Network Open. Protein Total 27 0.76 [0.73, 0.78] 67% 0.9 69% 71% 95% 23% 7.8% 3.6% 63% 26% Protein Total 27 0.76 [0.73, 0.78] 67% 0.8 63% 75% 87% 44% 15% 8.7% 58% 19% Folic Acid 26 0.63 [0.60, 0.66] 31% 0.95 80% 34% 88% 23% 16% 3.9% 28% 53% Folic Acid 26 0.63 [0.60, 0.66] 31% 0.9 78% 39% 71% 48% 33% 9.2% 22% 35% Folic Acid 26 0.63 [0.60, 0.66] 31% 0.8 73% 47% 31% 84% 58% 22% 9.6% 11% Folic Acid 26 0.63 [0.60, 0.66] 31% 0.99 77% 32% 99% 1.0% 0.70% 0.21% 31% 68% Glucose 26 0.74 [0.69, 0.79] 9.5% 0.8 91% - 0 100% 91% 9.5% 0 0 Glucose 26 0.74 [0.69, 0.79] 9.5% 0.9 91% - 0 100% 91% 9.5% 0 0 Glucose 26 0.74 [0.69, 0.79] 9.5% 0.95 93% 61% 28% 98% 89% 6.8% 2.6% 1.7% Glucose 26 0.74 [0.69, 0.79] 9.5% 0.99 94% 22% 55% 79% 72% 4.3% 5.2% 19% Respiratory Culture 24 0.67 [0.64, 0.71] 33% 0.99 82% 33% 98% 5.5% 3.7% 0.81% 32% 64% Respiratory Culture 24 0.67 [0.64, 0.70] 33% 0.95 80% 39% 80% 38% 26% 6.4% 26% 42% Respiratory Culture 24 0.67 [0.64, 0.70] 33% 0.9 78% 44% 64% 62% 42% 12% 21% 26% Respiratory Culture 24 0.67 [0.64, 0.70] 33% 0.8 73% 60% 31% 90% 61% 22% 10% 6.6% CSF Culture And Gram 23 0.6 [0.48, 0.72] 3.7% 0.8 96% - 0 100% 96% 3.7% 0 0 Stain CSF Culture And Gram 23 0.6 [0.49, 0.72] 3.7% 0.95 96% - 0 100% 96% 3.7% 0 0 Stain CSF Culture And Gram 23 0.6 [0.48, 0.72] 3.7% 0.99 97% 10% 10% 97% 93% 3.3% 0.37% 3.3% Stain CSF Culture And Gram 23 0.6 [0.50, 0.71] 3.7% 0.9 96% - 0 100% 96% 3.7% 0 0 Stain Occult Bld 23 0.76 [0.71, 0.81] 76% 0.8 70% 79% 97% 19% 4.5% 1.9% 74% 19% Occult Bld 23 0.76 [0.71, 0.81] 76% 0.95 - 76% 100% 0 0 0 76% 24% Occult Bld 23 0.76 [0.71, 0.81] 76% 0.99 - 76% 100% 0 0 0 76% 24% Occult Bld 23 0.76 [0.71, 0.81] 76% 0.9 78% 77% 100% 5.1% 1.2% 0.35% 76% 23% iSTAT Creatinine 11 0.72 [0.69, 0.75] 37% 0.8 74% 53% 59% 69% 43% 15% 22% 20% iSTAT Creatinine 11 0.72 [0.69, 0.74] 37% 0.9 82% 46% 85% 40% 25% 5.7% 31% 37% iSTAT Creatinine 11 0.72 [0.69, 0.75] 37% 0.95 87% 43% 94% 26% 16% 2.3% 35% 47% iSTAT Creatinine 11 0.72 [0.69, 0.74] 37% 0.99 93% 40% 98% 14% 8.8% 0.71% 36% 54% iSTAT Cg4 10 0.76 [0.69, 0.84] 13% 0.8 87% - 0 100% 87% 13% 0 0 iSTAT Cg4 10 0.76 [0.68, 0.84] 13% 0.9 88% 100% 11% 100% 87% 12% 1.5% 0 iSTAT Cg4 10 0.76 [0.69, 0.84] 13% 0.95 91% 49% 42% 93% 81% 7.7% 5.6% 5.9% iSTAT Cg4 10 0.76 [0.69, 0.84] 13% 0.99 93% 25% 67% 69% 60% 4.4% 8.9% 27% Anti-hiv 1 0.58 [0.41, 0.75] 1.2% 0.99 99% 2.8% 44% 81% 80% 0.69% 0.54% 19% Anti-hiv 1 0.58 [0.42, 0.74] 1.2% 0.95 99% - 0 100% 99% 1.2% 0 0 Anti-hiv 1 0.58 [0.43, 0.75] 1.2% 0.9 99% - 0 100% 99% 1.2% 0 0 Anti-hiv 1 0.58 [0.41, 0.74] 1.2% 0.8 99% - 0 100% 99% 1.2% 0 0 Stool Culture 0 0.68 [0.57, 0.79] 9.0% 0.99 100% 9.3% 100% 3.4% 3.1% 0 9.0% 88% Stool Culture 0 0.68 [0.57, 0.79] 9.0% 0.95 95% 11% 78% 38% 34% 2.0% 7.0% 57% Stool Culture 0 0.68 [0.57, 0.79] 9.0% 0.9 92% 50% 8.7% 99% 90% 8.2% 0.78% 0.78% © 2019 Xu S et al. JAMA Network Open. Stool Culture 0 0.68 [0.57, 0.80] 9.0% 0.8 91% - 0 100% 91% 9.0% 0 0 eTable 3. Diagnostic Metrics for Common Stanford Standalone Labs Lab Test Vol C 95% CI Prev Target NPV Sens Spec NPV Potassium 9546 0.76 [0.73, 0.79] 13% 0.99 98% 13% 100% 2.5% 2.2% 0.04% 13% 85% Potassium 9546 0.76 [0.73, 0.79] 13% 0.8 87% - 0 100% 87% 13% 0 0 Potassium 9546 0.76 [0.73, 0.79] 13% 0.9 89% 48% 19% 97% 84% 11% 2.4% 2.6% Potassium 9546 0.76 [0.73, 0.79] 13% 0.95 95% 25% 77% 66% 57% 3.0% 10% 30% Hemoglobin 8016 0.94 [0.92, 0.95] 88% 0.99 85% 89% 100% 7.9% 0.93% 0.17% 88% 11% Hemoglobin 8016 0.94 [0.92, 0.95] 88% 0.95 81% 90% 99% 17% 2.1% 0.50% 88% 9.8% Hemoglobin 8016 0.94 [0.92, 0.95] 88% 0.9 77% 92% 99% 36% 4.3% 1.3% 87% 7.6% Hemoglobin 8016 0.94 [0.92, 0.95] 88% 0.8 69% 95% 96% 59% 6.9% 3.2% 85% 4.9% Sodium 7289 0.87 [0.85, 0.88] 42% 0.99 98% 42% 100% 3.0% 1.7% 0.04% 42% 57% Sodium 7289 0.87 [0.85, 0.88] 42% 0.8 75% 87% 56% 94% 55% 18% 23% 3.5% Sodium 7289 0.87 [0.85, 0.88] 42% 0.9 86% 68% 84% 71% 41% 6.5% 35% 17% Sodium 7289 0.87 [0.85, 0.88] 42% 0.95 92% 53% 95% 41% 24% 2.1% 40% 34% Creatinine 7012 0.96 [0.96, 0.97] 40% 0.99 99% 63% 99% 61% 36% 0.49% 40% 23% Creatinine 7012 0.96 [0.96, 0.97] 40% 0.95 94% 79% 93% 83% 50% 2.9% 37% 10.0% Creatinine 7012 0.96 [0.96, 0.97] 40% 0.9 90% 89% 85% 93% 56% 6.0% 34% 4.1% Creatinine 7012 0.96 [0.96, 0.97] 40% 0.8 80% 98% 64% 99% 59% 15% 26% 0.56% Urea Nitrogen 6999 0.95 [0.94, 0.96] 30% 0.8 81% 96% 45% 99% 69% 16% 13% 0.63% Urea Nitrogen 6999 0.95 [0.94, 0.96] 30% 0.9 89% 90% 73% 97% 68% 8.0% 22% 2.4% Urea Nitrogen 6999 0.95 [0.94, 0.96] 30% 0.99 99% 42% 99% 41% 29% 0.27% 30% 41% Urea Nitrogen 6999 0.95 [0.93, 0.96] 30% 0.95 94% 77% 87% 89% 62% 4.0% 26% 7.9% Calcium 6963 0.89 [0.87, 0.90] 59% 0.99 100% 60% 100% 3.9% 1.6% 0 59% 39% Calcium 6963 0.89 [0.87, 0.90] 59% 0.95 92% 65% 99% 24% 9.8% 0.80% 58% 31% Calcium 6963 0.89 [0.87, 0.90] 59% 0.9 88% 71% 96% 45% 18% 2.6% 57% 23% Calcium 6963 0.89 [0.87, 0.90] 59% 0.8 79% 82% 87% 73% 30% 7.9% 51% 11% CO2 6929 0.86 [0.84, 0.88] 19% 0.99 95% 35% 88% 60% 48% 2.3% 17% 32% CO2 6929 0.86 [0.84, 0.88] 19% 0.95 93% 54% 72% 85% 69% 5.5% 14% 12% CO2 6929 0.86 [0.85, 0.88] 19% 0.9 88% 83% 47% 98% 79% 10% 9.0% 1.9% CO2 6929 0.86 [0.84, 0.88] 19% 0.8 81% - 0 100% 81% 19% 0 0 Platelet Count 6660 0.91 [0.90, 0.92] 45% 0.8 76% 88% 64% 93% 51% 16% 29% 3.8% Platelet Count 6660 0.91 [0.90, 0.92] 45% 0.9 89% 80% 88% 82% 45% 5.6% 40% 9.8% Platelet Count 6660 0.91 [0.90, 0.92] 45% 0.95 92% 72% 93% 70% 39% 3.1% 42% 16% Platelet Count 6660 0.91 [0.90, 0.92] 45% 0.99 98% 59% 99% 42% 23% 0.38% 45% 32% © 2019 Xu S et al. JAMA Network Open. White Blood 6422 0.89 [0.88, 0.91] 45% 0.99 100% 45% 100% 0.54% 0.30% 0 45% 55% Cells White Blood 6422 0.89 [0.88, 0.91] 45% 0.95 93% 60% 95% 48% 27% 2.1% 42% 29% Cells White Blood 6422 0.89 [0.88, 0.90] 45% 0.9 87% 70% 87% 70% 39% 5.9% 39% 16% Cells White Blood 6422 0.89 [0.88, 0.91] 45% 0.8 78% 88% 67% 93% 51% 15% 30% 4.0% Cells Albumin 3063 0.93 [0.91, 0.94] 84% 0.99 - 84% 100% 0 0 0 84% 16% Albumin 3063 0.93 [0.91, 0.94] 84% 0.8 - 84% 100% 0 0 0 84% 16% Albumin 3063 0.93 [0.91, 0.94] 84% 0.9 - 84% 100% 0 0 0 84% 16% Albumin 3063 0.93 [0.91, 0.94] 84% 0.95 - 84% 100% 0 0 0 84% 16% Total Bilirubin 3038 0.97 [0.96, 0.98] 29% 0.9 87% 99% 65% 100% 71% 10% 19% 0.14% Total Bilirubin 3038 0.97 [0.96, 0.97] 29% 0.8 74% 100% 14% 100% 71% 25% 4.0% 0 Total Bilirubin 3038 0.97 [0.96, 0.98] 29% 0.99 98% 58% 96% 72% 51% 1.1% 28% 20% Total Bilirubin 3038 0.97 [0.96, 0.98] 29% 0.95 94% 92% 84% 97% 69% 4.8% 24% 2.0% Protein 2993 0.91 [0.89, 0.92] 31% 0.8 83% 90% 57% 97% 67% 13% 18% 2.0% Protein 2993 0.91 [0.89, 0.92] 31% 0.9 90% 74% 78% 88% 60% 6.8% 24% 8.6% Protein 2993 0.91 [0.89, 0.92] 31% 0.95 93% 59% 89% 72% 50% 3.5% 28% 19% Protein 2993 0.91 [0.90, 0.92] 31% 0.99 97% 39% 98% 31% 21% 0.72% 31% 48% AST (SGOT) 2986 0.92 [0.91, 0.93] 35% 0.95 94% 61% 92% 69% 45% 2.8% 32% 21% AST (SGOT) 2986 0.92 [0.91, 0.93] 35% 0.99 97% 44% 98% 34% 22% 0.70% 34% 43% AST (SGOT) 2986 0.92 [0.91, 0.93] 35% 0.9 89% 78% 80% 88% 57% 6.8% 28% 8.0% AST (SGOT) 2986 0.92 [0.91, 0.93] 35% 0.8 81% 96% 57% 99% 65% 15% 20% 0.83% ALT (SGPT) 2986 0.93 [0.92, 0.94] 30% 0.99 98% 45% 98% 47% 33% 0.74% 30% 37% ALT (SGPT) 2986 0.93 [0.92, 0.94] 30% 0.95 93% 71% 85% 85% 59% 4.7% 26% 11% ALT (SGPT) 2986 0.93 [0.92, 0.94] 30% 0.9 89% 94% 71% 98% 68% 8.8% 22% 1.5% ALT (SGPT) 2986 0.93 [0.92, 0.94] 30% 0.8 76% 99% 30% 100% 69% 21% 9.0% 0.12% Alk Phos 2984 0.94 [0.93, 0.95] 45% 0.8 69% 99% 45% 100% 55% 25% 20% 0.22% Alk Phos 2984 0.94 [0.93, 0.95] 45% 0.95 90% 77% 89% 78% 43% 4.8% 40% 12% Alk Phos 2984 0.94 [0.93, 0.95] 45% 0.99 97% 60% 98% 47% 26% 0.84% 44% 29% Alk Phos 2984 0.94 [0.93, 0.95] 45% 0.9 86% 91% 81% 93% 52% 8.7% 36% 3.7% eTable 4. Diagnostic Metrics for Common Stanford Components Lab Test C 95% CI Prev Target NPV Sens Spec NPV Magnesium 0.83 [0.80, 0.86] 9.1% 0.99 97% 16% 81% 59% 53% 1.7% 7.4% 38% Magnesium 0.83 [0.79, 0.86] 9.1% 0.95 94% 61% 41% 97% 89% 5.4% 3.7% 2.3% Magnesium 0.83 [0.80, 0.86] 9.1% 0.9 91% - 0 100% 91% 9.1% 0 0 Magnesium 0.83 [0.79, 0.86] 9.1% 0.8 91% - 0 100% 91% 9.1% 0 0 © 2019 Xu S et al. JAMA Network Open. Phosphorus 0.75 [0.73, 0.77] 31% 0.99 95% 33% 99% 6.5% 4.5% 0.23% 31% 64% Phosphorus 0.75 [0.73, 0.78] 31% 0.95 90% 36% 94% 25% 17% 2.0% 29% 51% Phosphorus 0.75 [0.73, 0.78] 31% 0.9 86% 43% 81% 52% 35% 5.9% 25% 33% Phosphorus 0.75 [0.73, 0.77] 31% 0.8 78% 62% 46% 87% 60% 17% 15% 8.9% Hemoglobin A1c 0.69 [0.64, 0.73] 71% 0.99 87% 72% 100% 6.9% 2.0% 0.31% 70% 27% Hemoglobin A1c 0.69 [0.64, 0.73] 71% 0.95 57% 74% 94% 18% 5.3% 4.0% 67% 24% Hemoglobin A1c 0.69 [0.64, 0.73] 71% 0.9 52% 74% 91% 24% 7.0% 6.3% 64% 22% Hemoglobin A1c 0.69 [0.64, 0.73] 71% 0.8 50% 77% 84% 39% 11% 11% 59% 18% Uric Acid 0.88 [0.84, 0.92] 38% 0.99 96% 45% 98% 25% 16% 0.64% 38% 46% Uric Acid 0.88 [0.84, 0.92] 38% 0.95 92% 63% 91% 67% 41% 3.5% 35% 20% Uric Acid 0.88 [0.84, 0.92] 38% 0.9 86% 72% 79% 81% 50% 8.0% 30% 12% Uric Acid 0.88 [0.84, 0.92] 38% 0.8 80% 88% 61% 95% 58% 15% 23% 3.2% Albumin 0.76 [0.65, 0.86] 69% 0.99 50% 70% 96% 8.3% 2.6% 2.6% 66% 29% Albumin 0.76 [0.65, 0.86] 69% 0.95 50% 72% 89% 25% 7.8% 7.8% 61% 23% Albumin 0.76 [0.65, 0.86] 69% 0.9 54% 79% 79% 54% 17% 14% 55% 14% Albumin 0.76 [0.66, 0.86] 69% 0.8 45% 84% 58% 75% 23% 29% 40% 7.8% Thyroid Stimulating 0.67 [0.63, 0.70] 15% 0.99 92% 17% 87% 27% 23% 1.9% 13% 62% Hormone Thyroid Stimulating 0.67 [0.63, 0.71] 15% 0.95 89% 24% 46% 75% 64% 8.0% 6.7% 21% Hormone Thyroid Stimulating 0.67 [0.62, 0.71] 15% 0.9 88% 47% 24% 95% 81% 11% 3.6% 4.1% Hormone Thyroid Stimulating 0.67 [0.63, 0.71] 15% 0.8 85% - 0 100% 85% 15% 0 0 Hormone Troponin I 0.95 [0.94, 0.96] 32% 0.99 96% 62% 94% 73% 50% 1.8% 30% 18% Troponin I 0.95 [0.94, 0.96] 32% 0.95 93% 80% 85% 90% 61% 4.9% 27% 7.0% Troponin I 0.95 [0.94, 0.96] 32% 0.9 90% 90% 77% 96% 65% 7.5% 25% 2.7% Troponin I 0.95 [0.94, 0.96] 32% 0.8 80% 100% 46% 100% 68% 17% 15% 0 Potassium 0.7 [0.63, 0.78] 38% 0.99 84% 45% 90% 33% 21% 3.9% 34% 42% Potassium 0.7 [0.62, 0.78] 38% 0.95 76% 52% 69% 61% 38% 12% 26% 24% Potassium 0.7 [0.62, 0.77] 38% 0.9 74% 60% 55% 77% 48% 17% 21% 14% Potassium 0.7 [0.62, 0.78] 38% 0.8 68% 66% 28% 91% 57% 27% 11% 5.6% Sodium 0.94 [0.90, 0.98] 51% 0.99 86% 94% 85% 95% 46% 7.6% 43% 2.5% Sodium 0.94 [0.90, 0.98] 51% 0.95 83% 98% 80% 99% 48% 10% 41% 0.64% Sodium 0.94 [0.90, 0.98] 51% 0.9 72% 98% 64% 99% 48% 18% 32% 0.64% Sodium 0.94 [0.90, 0.98] 51% 0.8 59% 97% 35% 99% 48% 33% 18% 0.64% Calcium 0.81 [0.69, 0.91] 63% 0.99 64% 95% 69% 93% 34% 20% 44% 2.4% Calcium 0.81 [0.71, 0.91] 63% 0.95 64% 95% 69% 93% 34% 20% 44% 2.4% Calcium 0.81 [0.69, 0.91] 63% 0.9 64% 95% 69% 93% 34% 20% 44% 2.4% Calcium 0.81 [0.71, 0.91] 63% 0.8 64% 95% 69% 93% 34% 20% 44% 2.4% © 2019 Xu S et al. JAMA Network Open. eTable 5. Diagnostic Metrics for Top UMich Standalone Labs Lab Test C 95% CI Prev Target NPV Sens Spec NPV White Blood 0.83 [0.81, 0.84] 46% 0.99 94% 51% 99% 19% 10% 0.66% 45% 44% Cells White Blood 0.83 [0.81, 0.84] 46% 0.95 89% 55% 95% 34% 18% 2.2% 44% 36% Cells White Blood 0.83 [0.81, 0.85] 46% 0.9 84% 61% 88% 53% 28% 5.6% 41% 25% Cells White Blood 0.83 [0.81, 0.85] 46% 0.8 75% 74% 70% 78% 42% 14% 32% 12% Cells Hemoglobin 0.94 [0.93, 0.95] 77% 0.99 85% 79% 99% 14% 3.2% 0.58% 76% 20% Hemoglobin 0.94 [0.93, 0.95] 77% 0.95 82% 81% 99% 22% 5.0% 1.1% 76% 18% Hemoglobin 0.94 [0.93, 0.95] 77% 0.9 80% 83% 98% 33% 7.7% 1.9% 75% 15% Hemoglobin 0.94 [0.93, 0.95] 77% 0.8 73% 89% 93% 62% 14% 5.2% 72% 8.9% Platelet Count 0.92 [0.90, 0.93] 32% 0.99 98% 41% 98% 34% 23% 0.48% 31% 45% Platelet Count 0.92 [0.91, 0.93] 32% 0.95 93% 66% 88% 79% 54% 3.9% 28% 14% Platelet Count 0.92 [0.90, 0.93] 32% 0.9 89% 83% 74% 93% 64% 8.1% 23% 4.6% Platelet Count 0.92 [0.90, 0.93] 32% 0.8 79% 98% 41% 100% 68% 18% 13% 0.29% Sodium 0.92 [0.90, 0.93] 22% 0.99 - 22% 100% 0 0 0 22% 78% Sodium 0.92 [0.90, 0.93] 22% 0.95 96% 50% 88% 76% 60% 2.5% 19% 19% Sodium 0.92 [0.90, 0.93] 22% 0.9 93% 84% 72% 96% 76% 6.1% 15% 2.9% Sodium 0.92 [0.90, 0.93] 22% 0.8 78% - 0 100% 78% 22% 0 0 Potassium 0.76 [0.74, 0.79] 13% 0.99 100% 13% 100% 2.1% 1.8% 0 13% 85% Potassium 0.76 [0.74, 0.79] 13% 0.95 95% 21% 83% 54% 47% 2.3% 11% 40% Potassium 0.76 [0.74, 0.79] 13% 0.9 91% 40% 36% 92% 80% 8.3% 4.8% 7.0% Potassium 0.76 [0.74, 0.79] 13% 0.8 87% - 0 100% 87% 13% 0 0 Creatinine 0.9 [0.89, 0.91] 41% 0.99 95% 50% 97% 32% 19% 1.0% 40% 40% Creatinine 0.9 [0.89, 0.91] 41% 0.95 91% 63% 91% 62% 36% 3.7% 38% 22% Creatinine 0.9 [0.88, 0.91] 41% 0.9 87% 77% 82% 83% 48% 7.4% 34% 10% Creatinine 0.9 [0.89, 0.91] 41% 0.8 77% 92% 60% 96% 56% 16% 25% 2.2% Total Bilirubin 0.93 [0.91, 0.94] 28% 0.99 94% 63% 86% 81% 58% 3.9% 24% 14% Total Bilirubin 0.93 [0.91, 0.94] 28% 0.95 92% 87% 79% 96% 69% 5.8% 22% 3.1% Total Bilirubin 0.93 [0.91, 0.94] 28% 0.9 88% 99% 65% 100% 72% 9.7% 18% 0.23% Total Bilirubin 0.93 [0.91, 0.94] 28% 0.8 77% 100% 22% 100% 72% 22% 6.0% 0 CO2 0.87 [0.84, 0.89] 15% 0.99 100% 15% 100% 0.37% 0.31% 0 15% 85% CO2 0.87 [0.84, 0.89] 15% 0.95 94% 52% 68% 89% 76% 4.9% 10% 9.3% CO2 0.87 [0.85, 0.89] 15% 0.9 88% 83% 24% 99% 84% 11% 3.7% 0.74% © 2019 Xu S et al. JAMA Network Open. CO2 0.87 [0.85, 0.89] 15% 0.8 85% - 0 100% 85% 15% 0 0 AST (SGOT) 0.88 [0.86, 0.89] 48% 0.99 95% 50% 100% 6.8% 3.5% 0.20% 48% 48% AST (SGOT) 0.88 [0.87, 0.89] 48% 0.95 93% 56% 98% 29% 15% 1.1% 47% 37% AST (SGOT) 0.88 [0.87, 0.89] 48% 0.9 90% 61% 94% 45% 23% 2.6% 45% 28% AST (SGOT) 0.88 [0.87, 0.89] 48% 0.8 79% 79% 77% 81% 42% 11% 37% 9.8% ALT (SGPT) 0.92 [0.91, 0.93] 40% 0.99 99% 45% 100% 21% 13% 0.19% 40% 48% ALT (SGPT) 0.92 [0.91, 0.93] 40% 0.95 96% 59% 96% 56% 34% 1.5% 38% 26% ALT (SGPT) 0.92 [0.91, 0.93] 40% 0.9 90% 68% 88% 73% 44% 4.7% 35% 17% ALT (SGPT) 0.92 [0.91, 0.93] 40% 0.8 82% 95% 68% 97% 59% 13% 27% 1.6% Albumin 0.9 [0.89, 0.92] 43% 0.99 97% 49% 99% 20% 12% 0.32% 43% 45% Albumin 0.9 [0.89, 0.92] 43% 0.95 93% 56% 96% 43% 24% 1.9% 42% 32% Albumin 0.9 [0.89, 0.92] 43% 0.9 87% 73% 86% 75% 43% 6.3% 37% 14% Albumin 0.9 [0.89, 0.92] 43% 0.8 77% 93% 62% 96% 55% 16% 27% 2.0% Calcium 0.89 [0.88, 0.90] 35% 0.99 98% 40% 99% 20% 13% 0.20% 35% 52% Calcium 0.89 [0.88, 0.90] 35% 0.95 93% 55% 92% 59% 39% 2.8% 32% 27% Calcium 0.89 [0.88, 0.90] 35% 0.9 89% 68% 81% 80% 52% 6.7% 28% 13% Calcium 0.89 [0.88, 0.90] 35% 0.8 81% 88% 57% 96% 62% 15% 20% 2.8% Protein 0.91 [0.90, 0.92] 43% 0.99 99% 46% 100% 11% 6.0% 0.07% 43% 51% Protein 0.91 [0.90, 0.92] 43% 0.95 93% 55% 96% 39% 22% 1.6% 42% 34% Protein 0.91 [0.90, 0.92] 43% 0.9 89% 76% 87% 80% 45% 5.5% 38% 12% Protein 0.91 [0.90, 0.92] 43% 0.8 79% 94% 66% 97% 55% 15% 28% 1.9% Alk Phos 0.92 [0.91, 0.93] 27% 0.99 99% 39% 99% 43% 31% 0.27% 27% 42% Alk Phos 0.92 [0.91, 0.93] 27% 0.95 94% 58% 87% 77% 56% 3.5% 24% 17% Alk Phos 0.92 [0.91, 0.93] 27% 0.9 90% 83% 71% 94% 69% 7.8% 20% 4.0% Alk Phos 0.92 [0.91, 0.93] 27% 0.8 83% 99% 48% 100% 73% 14% 13% 0.12% Urea Nitrogen 0.93 [0.92, 0.94] 48% 0.99 97% 55% 99% 26% 13% 0.41% 48% 39% Urea Nitrogen 0.93 [0.92, 0.94] 48% 0.95 93% 69% 95% 60% 31% 2.2% 46% 21% Urea Nitrogen 0.93 [0.92, 0.94] 48% 0.9 89% 81% 90% 80% 42% 5.0% 43% 10% Urea Nitrogen 0.93 [0.92, 0.94] 48% 0.8 78% 94% 71% 96% 50% 14% 34% 2.3% eTable 6. Diagnostic Metrics for Common UMich Components Lab Test C 95% CI Prev Target NPV Sens Spec NPV Magnesium 0.81 [0.79, 0.83] 21% 0.99 - 21% 100% 0 0 0 21% 79% Magnesium 0.81 [0.78, 0.83] 21% 0.95 95% 32% 90% 48% 37% 2.1% 19% 41% Magnesium 0.81 [0.78, 0.83] 21% 0.9 90% 42% 71% 73% 58% 6.1% 15% 21% Magnesium 0.81 [0.79, 0.83] 21% 0.8 82% 79% 21% 98% 77% 17% 4.5% 1.2% Phosphorus 0.8 [0.78, 0.82] 24% 0.99 97% 26% 99% 9.4% 7.1% 0.24% 24% 69% Phosphorus 0.8 [0.78, 0.82] 24% 0.95 91% 39% 81% 60% 46% 4.5% 20% 30% © 2019 Xu S et al. JAMA Network Open. Phosphorus 0.8 [0.78, 0.82] 24% 0.9 88% 55% 62% 84% 64% 9.0% 15% 12% Phosphorus 0.8 [0.78, 0.82] 24% 0.8 79% 88% 15% 99% 75% 20% 3.7% 0.52% Prothrombin Time 0.93 [0.92, 0.94] 44% 0.99 97% 50% 99% 24% 13% 0.40% 44% 43% Prothrombin Time 0.93 [0.92, 0.94] 44% 0.95 92% 69% 93% 67% 38% 3.2% 41% 18% Prothrombin Time 0.93 [0.92, 0.94] 44% 0.9 87% 83% 84% 87% 49% 7.1% 37% 7.5% Prothrombin Time 0.93 [0.92, 0.94] 44% 0.8 78% 96% 66% 98% 55% 15% 29% 1.1% Partial Thromboplastin Time 0.92 [0.91, 0.93] 49% 0.99 93% 60% 97% 38% 20% 1.6% 48% 31% Partial Thromboplastin Time 0.92 [0.91, 0.93] 49% 0.95 89% 76% 91% 72% 36% 4.5% 45% 14% Partial Thromboplastin Time 0.92 [0.91, 0.93] 49% 0.9 86% 84% 86% 84% 43% 7.0% 42% 8.1% Partial Thromboplastin Time 0.92 [0.91, 0.93] 49% 0.8 81% 94% 77% 95% 48% 11% 38% 2.4% Alkaline Phosphatase 0.9 [0.89, 0.91] 46% 0.99 96% 55% 99% 30% 16% 0.64% 46% 37% Alkaline Phosphatase 0.9 [0.89, 0.91] 46% 0.95 93% 63% 95% 51% 27% 2.2% 44% 26% Alkaline Phosphatase 0.9 [0.89, 0.91] 46% 0.9 85% 72% 85% 71% 38% 6.8% 40% 16% Alkaline Phosphatase 0.9 [0.89, 0.91] 46% 0.8 78% 90% 70% 93% 50% 14% 32% 3.8% Sodium 0.87 [0.86, 0.89] 41% 0.99 - 41% 100% 0 0 0 41% 59% Sodium 0.87 [0.86, 0.89] 41% 0.95 92% 55% 94% 45% 26% 2.3% 39% 32% Sodium 0.87 [0.86, 0.89] 41% 0.9 87% 64% 86% 66% 39% 5.7% 36% 20% Sodium 0.87 [0.86, 0.89] 41% 0.8 78% 86% 62% 93% 54% 16% 26% 4.3% Potassium 0.76 [0.74, 0.79] 17% 0.99 96% 22% 94% 31% 26% 1.1% 16% 57% Potassium 0.76 [0.73, 0.79] 17% 0.95 92% 30% 71% 66% 55% 4.9% 12% 28% Potassium 0.76 [0.73, 0.79] 17% 0.9 88% 44% 37% 91% 75% 11% 6.2% 7.8% Potassium 0.76 [0.74, 0.79] 17% 0.8 83% - 0 100% 83% 17% 0 0 Troponin I 0.89 [0.88, 0.91] 44% 0.99 96% 48% 99% 15% 8.6% 0.36% 44% 47% Troponin I 0.89 [0.88, 0.91] 44% 0.95 91% 61% 94% 52% 29% 2.8% 42% 27% Troponin I 0.89 [0.88, 0.91] 44% 0.9 86% 71% 86% 72% 40% 6.3% 38% 16% Troponin I 0.89 [0.88, 0.91] 44% 0.8 80% 86% 72% 91% 50% 12% 32% 5.2% Lactate Dehydrogenase 0.94 [0.93, 0.95] 66% 0.99 97% 72% 100% 25% 8.2% 0.27% 66% 25% Lactate Dehydrogenase 0.94 [0.93, 0.95] 66% 0.95 94% 78% 99% 45% 15% 0.97% 65% 18% Lactate Dehydrogenase 0.94 [0.93, 0.95] 66% 0.9 87% 82% 96% 58% 20% 2.8% 64% 14% Lactate Dehydrogenase 0.94 [0.93, 0.95] 66% 0.8 76% 90% 87% 81% 27% 8.5% 58% 6.2% Calcium, Ionized 0.86 [0.84, 0.87] 63% 0.99 92% 64% 100% 3.7% 1.4% 0.11% 63% 36% Calcium, Ionized 0.86 [0.84, 0.87] 63% 0.95 89% 69% 98% 24% 9.0% 1.2% 62% 28% Calcium, Ionized 0.86 [0.84, 0.87] 63% 0.9 83% 75% 95% 45% 17% 3.4% 60% 20% Calcium, Ionized 0.86 [0.85, 0.87] 63% 0.8 74% 85% 84% 75% 28% 9.9% 53% 9.2% Uric Acid 0.91 [0.90, 0.93] 46% 0.99 97% 54% 99% 30% 17% 0.49% 45% 38% Uric Acid 0.91 [0.90, 0.93] 46% 0.95 94% 61% 97% 48% 26% 1.5% 44% 28% Uric Acid 0.91 [0.90, 0.93] 46% 0.9 90% 71% 90% 70% 38% 4.4% 41% 16% Uric Acid 0.91 [0.89, 0.93] 46% 0.8 83% 84% 79% 88% 48% 9.7% 36% 6.7% Albumin 0.85 [0.82, 0.87] 79% 0.99 100% 80% 100% 3.5% 0.73% 0 79% 20% © 2019 Xu S et al. JAMA Network Open. Albumin 0.85 [0.82, 0.87] 79% 0.95 54% 80% 99% 6.6% 1.4% 1.2% 78% 19% Albumin 0.85 [0.82, 0.87] 79% 0.9 58% 82% 97% 18% 3.8% 2.7% 77% 17% Albumin 0.85 [0.82, 0.87] 79% 0.8 56% 85% 92% 40% 8.3% 6.4% 73% 12% Thyroid Stimulating Hormone 0.66 [0.61, 0.71] 33% 0.99 80% 36% 88% 23% 15% 3.9% 29% 52% Thyroid Stimulating Hormone 0.66 [0.61, 0.71] 33% 0.95 76% 42% 65% 56% 37% 12% 21% 30% Thyroid Stimulating Hormone 0.66 [0.61, 0.71] 33% 0.9 74% 50% 42% 79% 53% 19% 14% 14% Thyroid Stimulating Hormone 0.66 [0.61, 0.71] 33% 0.8 71% 65% 19% 95% 64% 26% 6.4% 3.5% eTable 7. Diagnostic Metrics for Top UCSF Standalone Labs Lab Test C 95% CI Prev Target NPV Sens Spec NPV White Blood 0.89 [0.87, 0.90] 52% 0.99 94% 54% 99% 8.4% 4.0% 0.28% 51% 44% Cells White Blood 0.89 [0.87, 0.90] 52% 0.95 88% 63% 95% 41% 20% 2.6% 49% 28% Cells White Blood 0.89 [0.87, 0.90] 52% 0.9 85% 71% 90% 60% 29% 5.0% 47% 19% Cells White Blood 0.89 [0.87, 0.90] 52% 0.8 76% 86% 74% 88% 42% 14% 38% 6.0% Cells Hemoglobin 0.95 [0.94, 0.96] 89% 0.99 81% 90% 100% 4.3% 0.46% 0.11% 89% 10% Hemoglobin 0.95 [0.93, 0.96] 89% 0.95 76% 90% 100% 11% 1.1% 0.35% 89% 9.4% Hemoglobin 0.95 [0.94, 0.96] 89% 0.9 82% 92% 99% 25% 2.7% 0.60% 89% 7.8% Hemoglobin 0.95 [0.94, 0.96] 89% 0.8 73% 93% 98% 40% 4.3% 1.6% 88% 6.3% Platelet Count 0.95 [0.94, 0.96] 39% 0.99 97% 60% 98% 57% 35% 0.98% 38% 26% Platelet Count 0.95 [0.94, 0.96] 39% 0.95 92% 80% 89% 85% 52% 4.4% 35% 8.9% Platelet Count 0.95 [0.94, 0.96] 39% 0.9 88% 92% 79% 96% 58% 8.1% 31% 2.6% Platelet Count 0.95 [0.94, 0.96] 39% 0.8 76% 99% 50% 100% 60% 20% 20% 0.20% Sodium 0.89 [0.87, 0.90] 33% 0.99 100% 33% 100% 0.55% 0.37% 0 33% 67% Sodium 0.89 [0.88, 0.90] 33% 0.95 95% 49% 95% 53% 35% 1.7% 31% 32% Sodium 0.89 [0.88, 0.90] 33% 0.9 90% 64% 81% 78% 53% 6.1% 26% 15% Sodium 0.89 [0.87, 0.90] 33% 0.8 82% 86% 56% 96% 64% 14% 18% 3.0% Potassium 0.77 [0.75, 0.79] 16% 0.99 97% 20% 95% 28% 23% 0.84% 15% 61% Potassium 0.77 [0.74, 0.79] 16% 0.95 94% 28% 79% 61% 51% 3.4% 13% 33% Potassium 0.77 [0.74, 0.79] 16% 0.9 90% 41% 49% 87% 73% 8.3% 7.8% 11% Potassium 0.77 [0.74, 0.79] 16% 0.8 84% - 0 100% 84% 16% 0 0 Creatinine 0.94 [0.93, 0.95] 41% 0.99 96% 47% 99% 23% 14% 0.51% 40% 45% Creatinine 0.94 [0.93, 0.95] 41% 0.95 93% 72% 92% 76% 45% 3.2% 37% 14% Creatinine 0.94 [0.93, 0.95] 41% 0.9 89% 91% 83% 94% 56% 6.9% 34% 3.4% Creatinine 0.94 [0.93, 0.95] 41% 0.8 75% 98% 52% 99% 59% 19% 21% 0.47% Total Bilirubin 0.93 [0.91, 0.94] 30% 0.99 98% 44% 98% 48% 34% 0.70% 29% 37% © 2019 Xu S et al. JAMA Network Open. Total Bilirubin 0.93 [0.91, 0.94] 30% 0.95 93% 64% 87% 80% 56% 3.9% 26% 14% Total Bilirubin 0.93 [0.91, 0.94] 30% 0.9 89% 85% 72% 95% 67% 8.4% 21% 3.7% Total Bilirubin 0.93 [0.91, 0.94] 30% 0.8 81% 99% 45% 100% 70% 16% 13% 0.12% CO2 0.87 [0.86, 0.89] 21% 0.99 97% 24% 98% 13% 11% 0.37% 21% 68% CO2 0.87 [0.86, 0.89] 21% 0.95 95% 43% 86% 69% 55% 3.0% 18% 24% CO2 0.87 [0.86, 0.89] 21% 0.9 90% 67% 62% 92% 72% 8.1% 13% 6.6% CO2 0.87 [0.86, 0.89] 21% 0.8 79% 91% 1.7% 100% 79% 21% 0.37% 0.04% AST (SGOT) 0.85 [0.83, 0.86] 46% 0.99 94% 47% 100% 4.8% 2.6% 0.17% 46% 52% AST (SGOT) 0.85 [0.83, 0.86] 46% 0.95 90% 54% 96% 30% 16% 1.9% 44% 38% AST (SGOT) 0.85 [0.83, 0.86] 46% 0.9 86% 61% 90% 51% 28% 4.5% 41% 27% AST (SGOT) 0.85 [0.83, 0.86] 46% 0.8 77% 73% 72% 78% 42% 13% 33% 12% ALT (SGPT) 0.91 [0.90, 0.93] 31% 0.99 99% 41% 99% 36% 25% 0.35% 31% 44% ALT (SGPT) 0.91 [0.90, 0.93] 31% 0.95 93% 56% 89% 67% 46% 3.4% 28% 22% ALT (SGPT) 0.91 [0.90, 0.93] 31% 0.9 89% 80% 76% 91% 63% 7.7% 24% 6.0% ALT (SGPT) 0.91 [0.90, 0.93] 31% 0.8 78% 99% 39% 100% 69% 19% 12% 0.12% Albumin 0.91 [0.90, 0.92] 78% 0.99 100% 78% 100% 0.18% 0.04% 0 78% 22% Albumin 0.91 [0.90, 0.92] 78% 0.95 100% 78% 100% 0.18% 0.04% 0 78% 22% Albumin 0.91 [0.90, 0.92] 78% 0.9 79% 80% 99% 9.1% 2.0% 0.52% 78% 20% Albumin 0.91 [0.90, 0.92] 78% 0.8 75% 83% 97% 31% 6.8% 2.2% 76% 15% Calcium 0.88 [0.87, 0.89] 69% 0.99 88% 71% 99% 10% 3.2% 0.42% 69% 28% Calcium 0.88 [0.87, 0.90] 69% 0.95 88% 74% 99% 23% 7.2% 0.96% 68% 24% Calcium 0.88 [0.87, 0.90] 69% 0.9 85% 78% 97% 39% 12% 2.1% 67% 19% Calcium 0.88 [0.87, 0.90] 69% 0.8 75% 83% 91% 59% 18% 6.1% 63% 13% Protein 0.9 [0.89, 0.91] 54% 0.99 97% 57% 100% 12% 5.5% 0.16% 54% 40% Protein 0.9 [0.89, 0.91] 54% 0.95 93% 62% 98% 30% 14% 1.1% 53% 32% Protein 0.9 [0.89, 0.91] 54% 0.9 88% 69% 94% 51% 24% 3.2% 51% 22% Protein 0.9 [0.89, 0.91] 54% 0.8 81% 84% 83% 81% 37% 9.0% 45% 8.7% Alk Phos 0.93 [0.92, 0.94] 45% 0.99 98% 61% 99% 48% 26% 0.59% 45% 29% Alk Phos 0.93 [0.92, 0.94] 45% 0.95 92% 71% 93% 68% 37% 3.0% 42% 18% Alk Phos 0.93 [0.92, 0.94] 45% 0.9 87% 78% 86% 79% 43% 6.3% 39% 11% Alk Phos 0.93 [0.92, 0.94] 45% 0.8 79% 96% 69% 97% 53% 14% 31% 1.4% Urea Nitrogen 0.93 [0.91, 0.94] 38% 0.99 98% 47% 99% 31% 19% 0.38% 38% 43% Urea Nitrogen 0.93 [0.91, 0.94] 38% 0.95 92% 67% 90% 73% 45% 3.7% 34% 17% Urea Nitrogen 0.93 [0.91, 0.93] 38% 0.9 89% 81% 83% 88% 54% 6.6% 31% 7.5% Urea Nitrogen 0.93 [0.91, 0.94] 38% 0.8 79% 95% 58% 98% 61% 16% 22% 1.2% eTable 8. Diagnostic Metrics for Common UCSF Components Lab Test Stanford - Stanford - Stanford - UCSF - UCSF - UCSF - UMich - UMich - UMich - Stanford UCSF UMich Stanford UCSF UMich Stanford UCSF UMich © 2019 Xu S et al. JAMA Network Open. White Blood 0.89 0.88 0.79 0.87 0.88 0.81 0.87 0.88 0.83 Cells [0.88, 0.91] [0.86, 0.89] [0.77, 0.81] [0.86, 0.89] [0.87, 0.9] [0.8, 0.83] [0.86, 0.89] [0.86, 0.89] [0.81, 0.84] Hemoglobin 0.93 0.94 0.89 0.86 0.9 0.79 0.92 0.94 0.9 [0.92, 0.94] [0.93, 0.95] [0.88, 0.91] [0.84, 0.88] [0.89, 0.92] [0.77, 0.81] [0.9, 0.93] [0.93, 0.95] [0.89, 0.91] Platelet Count 0.91 0.94 0.91 0.89 0.95 0.91 0.89 0.94 0.92 [0.9, 0.92] [0.93, 0.95] [0.9, 0.93] [0.88, 0.91] [0.94, 0.96] [0.89, 0.92] [0.88, 0.91] [0.93, 0.95] [0.9, 0.93] Sodium 0.87 0.88 0.91 0.85 0.89 0.91 0.86 0.86 0.91 [0.85, 0.88] [0.86, 0.89] [0.9, 0.93] [0.84, 0.87] [0.87, 0.9] [0.89, 0.92] [0.84, 0.88] [0.84, 0.88] [0.9, 0.93] Potassium 0.76 0.75 0.67 0.74 0.77 0.75 0.73 0.75 0.76 [0.73, 0.79] [0.73, 0.78] [0.63, 0.7] [0.71, 0.77] [0.74, 0.79] [0.72, 0.78] [0.69, 0.76] [0.72, 0.77] [0.73, 0.79] CO2 0.86 0.8 0.75 0.8 0.87 0.85 0.77 0.82 0.87 [0.84, 0.88] [0.78, 0.82] [0.71, 0.78] [0.77, 0.82] [0.86, 0.89] [0.82, 0.87] [0.74, 0.8] [0.8, 0.84] [0.84, 0.88] Urea Nitrogen 0.95 0.92 0.9 0.94 0.92 0.9 0.93 0.92 0.92 [0.94, 0.96] [0.91, 0.93] [0.89, 0.92] [0.93, 0.95] [0.91, 0.93] [0.89, 0.92] [0.92, 0.94] [0.91, 0.93] [0.91, 0.93] Creatinine 0.96 0.91 0.85 0.94 0.94 0.88 0.92 0.88 0.9 [0.96, 0.97] [0.89, 0.92] [0.83, 0.86] [0.94, 0.95] [0.93, 0.95] [0.87, 0.9] [0.91, 0.93] [0.86, 0.89] [0.88, 0.91] Calcium 0.88 0.86 0.81 0.87 0.87 0.85 0.85 0.86 0.89 [0.87, 0.9] [0.85, 0.88] [0.79, 0.83] [0.85, 0.88] [0.86, 0.88] [0.83, 0.86] [0.83, 0.87] [0.84, 0.87] [0.88, 0.9] Albumin 0.92 0.88 0.73 0.84 0.89 0.74 0.92 0.89 0.9 [0.91, 0.93] [0.87, 0.9] [0.7, 0.75] [0.82, 0.86] [0.88, 0.9] [0.72, 0.76] [0.9, 0.93] [0.87, 0.9] [0.89, 0.92] Protein 0.91 0.89 0.87 0.88 0.89 0.85 0.89 0.88 0.9 [0.89, 0.92] [0.88, 0.9] [0.86, 0.88] [0.87, 0.9] [0.88, 0.9] [0.83, 0.86] [0.87, 0.9] [0.86, 0.89] [0.89, 0.91] Alk Phos 0.94 0.91 0.89 0.92 0.93 0.89 0.92 0.93 0.92 [0.93, 0.95] [0.9, 0.92] [0.88, 0.91] [0.91, 0.93] [0.92, 0.94] [0.87, 0.9] [0.91, 0.93] [0.92, 0.94] [0.91, 0.93] Total Bilirubin 0.96 0.91 0.91 0.95 0.93 0.91 0.96 0.92 0.92 [0.95, 0.97] [0.89, 0.92] [0.89, 0.93] [0.94, 0.97] [0.91, 0.94] [0.89, 0.92] [0.94, 0.97] [0.9, 0.93] [0.9, 0.93] AST (SGOT) 0.92 0.81 0.86 0.85 0.77 0.73 0.88 0.77 0.86 [0.91, 0.93] [0.8, 0.83] [0.85, 0.87] [0.83, 0.86] [0.76, 0.79] [0.71, 0.75] [0.87, 0.9] [0.75, 0.79] [0.85, 0.87] ALT (SGPT) 0.93 0.86 0.91 0.92 0.91 0.88 0.88 0.84 0.88 [0.92, 0.94] [0.84, 0.87] [0.9, 0.92] [0.91, 0.93] [0.9, 0.93] [0.86, 0.89] [0.87, 0.89] [0.82, 0.86] [0.87, 0.9] eTable 9. Diagnostic Metrics for Common Components in Transferability Study Lab Test Medicare Chargemaster Magnesium $8.27 $280.00 Prothrombin Time $4.85 $190.00 Phosphorus $5.85 $225.00 Partial Thromboplastin Time $7.98 $240.00 Lactate $11.87 $330.00 Calcium Ionized $13.73 $399.00 Potassium $5.68 $204.00 Troponin I $12.47 $520.00 LDH Total $6.71 $193.00 Heparin $16.16 $466.00 Urinalysis - $196.00 © 2019 Xu S et al. JAMA Network Open. Blood Culture (Aerobic & Anaerobic) - $499.00 Blood Culture (2 Aerobic) - $499.00 Sodium $5.94 $219.00 Lidocaine $18.14 $264.00 Hematocrit $2.93 $217.00 Urine Culture $9.96 $317.00 Urinalysis With Microscopic $3.76 $196.00 Uric Acid $5.58 $135.00 Hemoglobin A1c $11.99 $145.00 Sepsis Protocol Lactate $11.87 $330.00 iSTAT Troponin I - $520.00 Platelet Count $5.53 $138.00 Lipase $8.51 $213.00 Procalcitonin $33.08 $318.00 Lactic Acid $13.19 $330.00 Fibrinogen $14.02 $285.00 Thyroid Stimulating Hormone $20.75 $380.00 Creatine Kinase $8.04 $254.00 C-Reactive Protein $6.39 $179.00 -proBNP $41.90 $821.00 Triglycerides $7.09 $175.00 $14.26 $446.00 C.diff Toxin B Gene - $486.00 Osmolality $8.16 $209.00 Respiratory Culture And Gram Stain - $226.00 Sedimentation Rate (ESR) - $75.00 Ferritin $16.83 $375.00 Albumin - $138.00 Ammonia - $334.00 Specific Gravity $3.28 $115.00 Fungal Culture - $309.00 Haptoglobin - $258.00 Anaerobic Culture $11.66 $544.00 Fluid Culture And Gram Stain - $226.00 Cmv Dna Pcr Quant - $849.00 Blood Cult Central Line Catheter By Nurse - $499.00 - $246.00 Transferrin Saturation $15.76 $292.50 Vitamin B12 $18.61 $281.00 © 2019 Xu S et al. JAMA Network Open. Blood Cult - First Set - $499.00 Prealbumin $18.01 $246.00 Osmolality $8.42 $255.00 Digoxin - $389.00 Reticulocyte Count Automated $4.93 $179.00 Cortisol - $374.00 Hepatitis B Antigen $12.75 $220.00 Iron - $165.00 AFB Culture - $162.00 Calcium - $143.00 Biopsy/tissue With Gram Stain - $182.00 Afb Culture - $249.50 Pregnancy Test $8.61 $102.00 Urea Nitrogen - $157.00 Gram Stain - $150.00 - $747.00 Protein Total $21.24 $229.00 Folic Acid - $274.00 Glucose - $250.00 Respiratory Culture - $302.00 CSF Culture And Gram Stain - $226.00 Occult Bld - $130.00 iSTAT Creatinine $6.33 $179.00 iSTAT Cg4 - $572.50 Anti-hiv - $280.00 Stool Culture $11.66 $402.50 eTable 10. Medicare and Chargemaster Fees for Standalone Labs © 2019 Xu S et al. JAMA Network Open. lab feature 1 score 1 feature 2 score 2 feature 3 score 3 Hemoglobin A1c last_normality 0.437 Diabetes 0.154 HCT 0.054 AFB Culture last_normality 0.639 LABAFBC 0.132 Birth 0.087 Afb Culture AdmitDxDate 0.467 BP_Low_Diastolic 0.267 Pulse 0.2 Albumin ALB 0.434 last_normality 0.094 Temp 0.086 Anaerobic Culture last_normality 0.448 Temp 0.102 Pulse 0.078 Vitamin B12 ALB 0.136 TBIL 0.119 PLT 0.093 Blood Culture (Aerobic & Temp 0.178 Pulse 0.124 BP_High_Systolic 0.11 Anaerobic) Blood Culture (2 Aerobic) PLT 0.149 Pulse 0.119 last_normality 0.113 Blood Cult - First Set WBC 0.252 Temp 0.168 K 0.128 Blood Cult Central Line Catheter Pulse 0.211 Temp 0.168 TBIL 0.1 By Nurse Urea Nitrogen BUN 0.408 LABBUN 0.14 CR 0.067 Biopsy/tissue With Gram Stain last_normality 0.312 WBC 0.166 PLT 0.091 Calcium CA 0.634 Pulse 0.083 last_normality 0.08 Calcium Ionized CAION 0.344 last_normality 0.182 PHCAI 0.154 C.difficile Toxin B Gene LABCDTPCR 0.295 Pulse 0.137 last_normality 0.089 Creatine Kinase CK 0.411 last_normality 0.264 PHA 0.055 Cmv Dna Pcr Quant CMVLOG 0.266 last_normality 0.212 CMVCP 0.084 Cortisol K 0.187 LAC 0.121 CO2 0.117 C-Reactive Protein CRP 0.217 Temp 0.161 last_normality 0.152 CSF Culture And Gram Stain WBC 0.173 Pulse 0.14 Temp 0.139 Glucose GLUCSF 0.393 last_normality 0.109 NA 0.068 Protein Total TPCSF 0.191 Pulse 0.113 BP_High_Systolic 0.091 Digoxin last_normality 0.759 DIG 0.188 Pulse 0.024 Sedimentation Rate (ESR) HCT 0.269 last_normality 0.134 ALB 0.12 Fungal Culture last_normality 0.152 Resp 0.116 Pulse 0.111 Iron Total Temp 0.158 Male 0.125 PLT 0.121 Ferritin last_normality 0.345 ALB 0.175 Temp 0.072 Fibrinogen FIBRINOGEN 0.463 last_normality 0.237 PLT 0.046 Fluid Culture And Gram Stain Temp 0.175 SurgerySpecialty 0.134 Pulse 0.106 Folic Acid Birth 0.195 CA 0.129 BP_High_Systolic 0.103 © 2019 Xu S et al. JAMA Network Open. T4 Free BP_High_Systolic 0.155 last_normality 0.124 Temp 0.097 Gram Stain BP_High_Systolic 0.116 BP_Low_Diastolic 0.093 Resp 0.091 Haptoglobin last_normality 0.547 TBIL 0.08 PLT 0.058 Hepatitis B Antigen last_normality 0.302 AdmitDxDate 0.284 order_time 0.067 Hematocrit HCT 0.577 last_normality 0.114 Resp 0.061 Heparin last_normality 0.431 HEPAR 0.179 Resp 0.078 Anti-hiv PLT 0.255 HCT 0.196 Temp 0.101 Potassium K 0.678 last_normality 0.085 HCT 0.082 Lactic Acid LAC 0.504 last_normality 0.263 PHA 0.044 Lactate last_normality 0.707 LACWBL 0.138 Pulse 0.027 LDH Total last_normality 0.773 LDH 0.168 WBC 0.01 Lidocaine LIDO 0.329 last_normality 0.315 BP_Low_Diastolic 0.078 Lipase last_normality 0.66 Pulse 0.047 LIPASE 0.045 Ck Mb (mass) last_normality 0.31 CKMBRI 0.124 CKMB 0.109 Magnesium MG 0.471 last_normality 0.235 Resp 0.044 Sodium last_normality 0.659 NA 0.295 Pulse 0.009 Ammonia NH3 0.362 last_normality 0.202 PLT 0.076 Nt - Probnp last_normality 0.318 Birth 0.204 BUN 0.094 Osmolality NA 0.338 last_normality 0.303 OSMOL 0.276 Prealbumin PREALBUMIN 0.591 ALB 0.064 HCT 0.056 iSTAT Cg4 Pulse 0.273 BP_Low_Diastolic 0.176 Temp 0.129 iSTAT Creatinine CR 0.571 Birth 0.172 BP_High_Systolic 0.166 iSTAT Troponin I Comorbidity.MI 0.376 last_normality 0.161 TNI 0.155 Phosphorus last_normality 0.472 PHOS 0.153 CR 0.066 Platelet Count PLT 0.719 HCT 0.061 Temp 0.031 Procalcitonin CR 0.19 last_normality 0.139 PROCTL 0.101 Prothrombin Time PT 0.348 INR 0.262 last_normality 0.15 Teg PO2V 0.237 PCO2V 0.201 PO2A 0.156 Partial Thromboplastin Time last_normality 0.756 PTT 0.077 Pulse 0.032 Respiratory Culture last_normality 0.351 Urine 0.161 PCO2A 0.117 Respiratory Culture And Gram last_normality 0.359 Glasgow Coma 0.131 Resp 0.111 Stain Scale Score © 2019 Xu S et al. JAMA Network Open. Reticulocyte Count Automated last_normality 0.487 HCT 0.254 TBIL 0.076 Sepsis Protocol Lactate LACWBL 0.3 last_normality 0.125 Temp 0.103 Stool Culture BP_Low_Diastolic 0.161 Temp 0.108 Pulse 0.092 Occult Bld K 0.476 order_time 0.256 WBC 0.051 Troponin I TNI 0.442 last_normality 0.29 Pulse 0.045 Transferrin Saturation ALB 0.241 HCT 0.14 PLT 0.125 Triglycerides TGL 0.603 last_normality 0.244 Birth 0.017 Thyroid Stimulating Hormone last_normality 0.181 BP_High_Systolic 0.143 BP_Low_Diastolic 0.131 Osmolality K 0.254 CO2 0.189 BUN 0.187 Urinalysis With Microscopic Pulse 0.242 LABUA 0.142 AdmitDxDate 0.136 Urinalysis Pulse 0.128 BP_High_Systolic 0.125 Temp 0.095 Pregnancy Test AdmitDxDate 0.177 Pulse 0.167 BP_Low_Diastolic 0.155 Uric Acid URIC 0.26 last_normality 0.238 CR 0.161 Urine Culture Male 0.241 last_normality 0.142 Temp 0.131 Specific Gravity SPG 0.2 HCT 0.2 Temp 0.2 eTable 11: Top 3 Important Features for Stanford Standalone Labs lab feature 1 score feature 2 score feature 3 score 3 1 2 White Blood Cells WBC 0.558 last_normality 0.244 PLT 0.065 Hemoglobin HCT 0.287 HGB 0.217 last_normality 0.181 Platelet Count last_normality 0.665 PLT 0.238 Pulse 0.014 Sodium last_normality 0.787 NA 0.184 Resp 0.009 Potassium last_normality 0.576 K 0.271 Pulse 0.067 Creatinine last_normality 0.87 CR 0.1 Birth 0.005 Urea Nitrogen last_normality 0.809 BUN 0.089 CR 0.017 CO2 CO2 0.539 last_normality 0.141 Pulse 0.053 Calcium CA 0.532 last_normality 0.246 Pulse 0.042 Protein TP 0.677 last_normality 0.117 CA 0.033 Albumin ALB 0.459 last_normality 0.208 Pulse 0.107 © 2019 Xu S et al. JAMA Network Open. Alk Phos ALKP 0.672 last_normality 0.235 AdmitDxDate 0.016 Total Bilirubin TBIL 0.62 last_normality 0.307 PLT 0.019 AST (SGOT) AST 0.642 last_normality 0.217 Resp 0.028 ALT (SGPT) last_normality 0.848 ALT 0.115 ALB 0.008 eTable 12. Top 3 Important Features for Common Stanford Components lab feature 1 score feature 2 score feature 3 score 3 1 2 Magnesium last_normality 0.605 MAG 0.286 AdmitDxDate 0.033 Phosphorus PHOS 0.53 last_normality 0.154 CREAT 0.088 Hemoglobin A1c AdmitDxDate 0.171 order_time 0.141 Birth 0.099 Uric Acid last_normality 0.643 URIC 0.241 HCT 0.032 Albumin ALB 0.365 HCT 0.157 CAL 0.109 Thyroid Stimulating AdmitDxDate 0.173 order_time 0.139 Birth 0.122 Hormone Troponin I TROP 0.683 last_normality 0.151 AdmitDxDate 0.035 Potassium POT 0.364 Birth 0.116 AdmitDxDate 0.11 Sodium SOD 0.714 last_normality 0.082 POT 0.057 Calcium CAL 0.505 WBC 0.17 CREAT 0.105 eTable 13. Top 3 Important Features for Top UMich Standalone Labs lab feature 1 score feature 2 score feature 3 score 3 1 2 White Blood Cells last_normality 0.62 WBC 0.217 AdmitDxDate 0.048 Hemoglobin HGB 0.39 last_normality 0.338 HCT 0.209 Platelet Count last_normality 0.836 PLT 0.119 Birth 0.016 Sodium last_normality 0.837 SOD 0.11 AdmitDxDate 0.035 Potassium POT 0.667 last_normality 0.16 CREAT 0.063 Creatinine last_normality 0.881 CREAT 0.071 AdmitDxDate 0.017 © 2019 Xu S et al. JAMA Network Open. Total Bilirubin TBIL 0.475 last_normality 0.211 PLT 0.109 CO2 last_normality 0.755 CO2 0.174 AdmitDxDate 0.026 AST (SGOT) last_normality 0.753 AST 0.176 Birth 0.026 ALT (SGPT) last_normality 0.853 ALT 0.118 Birth 0.011 Albumin last_normality 0.802 ALB 0.074 Birth 0.054 Calcium CAL 0.622 last_normality 0.266 AdmitDxDate 0.021 Protein PROT 0.446 last_normality 0.222 HCT 0.059 Alk Phos last_normality 0.775 ALK 0.113 Birth 0.044 Urea Nitrogen last_normality 0.79 UN 0.116 Birth 0.034 eTable 14. Top 3 Important Features for Common UMich Components lab feature 1 score feature 2 score feature 3 score 3 1 2 Magnesium last_normality 0.585 MG 0.297 LABMGN 0.044 Phosphorus PO4 0.483 last_normality 0.169 CREAT 0.098 Prothrombin Time PT 0.46 INR 0.221 last_normality 0.152 Partial Thromboplastin Time PTT 0.593 last_normality 0.127 LABPTT 0.091 Alkaline Phosphatase ALKP 0.643 last_normality 0.183 Alkaline Phosphatase 0.038 Sodium NA 0.941 Pulse 0.015 last_normality 0.012 Potassium K 0.558 last_normality 0.168 LABK 0.067 Troponin I TRPI 0.302 last_normality 0.25 LABTNI 0.222 Lactate Dehydrogenase LD 0.595 last_normality 0.193 LABLDH 0.1 Calcium, Ionized CAI 0.529 last_normality 0.191 CA 0.172 Uric Acid URIC 0.629 last_normality 0.217 PLT 0.02 Albumin ALB 0.358 CA 0.187 HCT 0.107 Thyroid Stimulating Hormone last_normality 0.247 HCT 0.205 Temp 0.117 eTable 15. Top 3 Important Features for Top UCSF Standalone Labs © 2019 Xu S et al. JAMA Network Open. lab feature 1 score feature 2 score feature 3 score 3 1 2 White Blood Cells last_normality 0.719 WBC 0.216 Pulse 0.024 Hemoglobin last_normality 0.767 HGB 0.128 DBP 0.025 Platelet Count last_normality 0.886 PLT 0.091 SBP 0.007 Sodium last_normality 0.755 NA 0.191 CREAT 0.012 Potassium K 0.442 last_normality 0.118 SBP 0.072 Creatinine last_normality 0.836 CREAT 0.088 Comorbidity.RenalDisease 0.034 Total Bilirubin last_normality 0.791 TBILI 0.138 SBP 0.014 CO2 CO2 0.684 last_normality 0.218 HCT 0.025 AST (SGOT) last_normality 0.715 AST 0.196 Pulse 0.019 ALT (SGPT) ALT 0.587 last_normality 0.176 Pulse 0.074 Albumin ALB 0.502 last_normality 0.211 CA 0.078 Calcium CA 0.734 last_normality 0.125 Pulse 0.031 Protein TP 0.656 last_normality 0.137 DBP 0.048 Alk Phos last_normality 0.765 ALKP 0.175 SBP 0.01 Urea Nitrogen last_normality 0.745 BUN 0.139 Comorbidity.RenalDisease 0.018 eTable 16. Top 3 Important Features for Common UCSF Components © 2019 Xu S et al. JAMA Network Open. eMethods. Technical Details of Machine Learning Algorithm 1.Recursive feature elimination with cross validation (using a RandomForest estimator) We apply recursive feature elimination with cross validation (RFECV) to select top 5% relevant features for prediction as implemented by scikit-learn (http://scikit-learn.org/). This process uses all features in the development set to train a random forest prediction model, identifying which features appear the least important towards the predicted outcome as assessed by a Gini entropy score. We removed those least relevant features and repeated this process recursively, until there were only 5% of the total original features left. 2. Hyperparameter tuning for machine learning algorithms Eac that can tune the performance of the algorithms. For example, penalized logistic regression includes a regularization penalty that specifies how much the algorithm should balance using more information vs. developing the simplest model that uses the fewest number of features. Similarly, random forest models have a maximum tree depth hyperparameter that constrains how many features are considered when building decision trees, to balance the bias-variance tradeoff that can impact the generalizability of model accuracy. We systematically tested all models across a range of plausible hyperparameter values to identify the most effective choices (eTable 1). 3. Applying locally trained model to a remote dataset We apply pre-trained models to data of a remote site by manually mapping identical columns. to process the full UMich feature matrix with 603 columns. The template includes 43 columns imputed and selected (by RFECV) from Stanford training set. The same set of 43 features were then select from the UMich matrix and imputed accordingly if they exist. If instead, the UMich dataset does not have a corresponding feature (e.g. patient vitals), we will create a dummy column with the same feature name but filled with the corresponding constant imputation value. - ready to be fed into Stanford-trained model. © 2019 Xu S et al. JAMA Network Open.

Journal

JAMA Network OpenAmerican Medical Association

Published: Sep 11, 2019

There are no references for this article.