Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Exploiting Rules to Enhance Machine Learning in Extracting Information From Multi-Institutional Prostate Pathology Reports

Exploiting Rules to Enhance Machine Learning in Extracting Information From Multi-Institutional... PURPOSE: Literature on clinical note mining has highlighted the superiority of machine learning (ML) over hand-crafted rules. Nevertheless, most studies assume the availability of large training sets, which is rarely the case. For this reason, in the clinical setting, rules are still common. We suggest 2 methods to leverage the knowledge encoded in pre-existing rules to inform ML decisions and obtain high performance, even with scarce annotations. METHODS: We collected 501 prostate pathology reports from 6 American hospitals. Reports were split into 2,711 core segments, annotated with 20 attributes describing the histology, grade, extension, and location of tumors. The data set was split by institutions to generate a cross-institutional evaluation setting. We assessed 4 systems, namely a rule-based approach, an ML model, and 2 hybrid systems integrating the previous methods: a Rule as Feature model and a Classifier Confidence model. Several ML algorithms were tested, including logistic regression (LR), support vector machine (SVM), and eXtreme gradient boosting (XGB). RESULTS: When training on data from a single institution, LR lags behind the rules by 3.5% (F1 score: 92.2% v 95.7%). Hybrid models, instead, obtain competitive results, with Classifier Confidence outperforming the rules by +0.5% (96.2%). When a larger amount of data from multiple institutions is used, LR improves by +1.5% over the rules (97.2%), whereas hybrid systems obtain +2.2% for Rule as Feature (97.7%) and +2.6% for Classifier Confidence (98.3%). Replacing LR with SVM or XGB yielded similar performance gains. CONCLUSION: We developed methods to use pre-existing handcrafted rules to inform ML algorithms. These hybrid systems obtain better performance than either rules or ML models alone, even when training data are limited. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JCO Clinical Cancer Informatics Wolters Kluwer Health

Exploiting Rules to Enhance Machine Learning in Extracting Information From Multi-Institutional Prostate Pathology Reports

Download PDF
10 pages

Loading next page...
 
/lp/wolters-kluwer-health/exploiting-rules-to-enhance-machine-learning-in-extracting-information-TKMJJWj6y2
Publisher
Wolters Kluwer Health
Copyright
(C) 2020 American Society of Clinical Oncology
ISSN
2473-4276
DOI
10.1200/CCI.20.00028
Publisher site
See Article on Publisher Site

Abstract

PURPOSE: Literature on clinical note mining has highlighted the superiority of machine learning (ML) over hand-crafted rules. Nevertheless, most studies assume the availability of large training sets, which is rarely the case. For this reason, in the clinical setting, rules are still common. We suggest 2 methods to leverage the knowledge encoded in pre-existing rules to inform ML decisions and obtain high performance, even with scarce annotations. METHODS: We collected 501 prostate pathology reports from 6 American hospitals. Reports were split into 2,711 core segments, annotated with 20 attributes describing the histology, grade, extension, and location of tumors. The data set was split by institutions to generate a cross-institutional evaluation setting. We assessed 4 systems, namely a rule-based approach, an ML model, and 2 hybrid systems integrating the previous methods: a Rule as Feature model and a Classifier Confidence model. Several ML algorithms were tested, including logistic regression (LR), support vector machine (SVM), and eXtreme gradient boosting (XGB). RESULTS: When training on data from a single institution, LR lags behind the rules by 3.5% (F1 score: 92.2% v 95.7%). Hybrid models, instead, obtain competitive results, with Classifier Confidence outperforming the rules by +0.5% (96.2%). When a larger amount of data from multiple institutions is used, LR improves by +1.5% over the rules (97.2%), whereas hybrid systems obtain +2.2% for Rule as Feature (97.7%) and +2.6% for Classifier Confidence (98.3%). Replacing LR with SVM or XGB yielded similar performance gains. CONCLUSION: We developed methods to use pre-existing handcrafted rules to inform ML algorithms. These hybrid systems obtain better performance than either rules or ML models alone, even when training data are limited.

Journal

JCO Clinical Cancer Informatics – Wolters Kluwer Health

Published: Oct 2, 2020

References

  • The potential for artificial intelligence in healthcare
    Davenport T; Kalakota R
  • The feasibility of using natural language processing to extract clinical information from breast pathology reports
    Buckley JM; Coopey SB; Sharko J
  • Mining fall-related information in clinical notes: Comparison of rule-based and novel word embedding-based machine learning approaches
    Topaz M; Murga L; Gaddis KM
  • Format
    Format
  • Information extraction from multi-institutional radiology reports
    Hassanpour S; Langlotz CP
  • Automating the determination of prostate cancer risk strata from electronic medical records
    Gregg JR; Lang M; Wang LL
  • Natural language processing in pathology: A scoping review
    Burger G; Abu-Hanna A; de Keizer N
  • Artificial intelligence-driven structurization of diagnostic information in free-text pathology reports
    Giannaris PS; Al-Taie Z; Kovalenko M
  • Automated extraction of grade, stage, and quality information from transurethral resection of bladder tumor pathology reports using natural language processing
    Glaser AP; Jordan BJ; Cohen J
  • From big data to precision medicine
    Hulsen T; Jamuar SS; Moody AR
  • Text mining for precision medicine: Bringing structure to EHRs and biomedical literature to understand genes and health
    Simmons M; Singhal A; Lu Z
  • Rule-Based Information Extraction is Dead! Long Live Rule-Based Information Extraction Systems! EMNLP 2013;:pp
    Chiticariu L; Li Y; Reiss F
  • Using machine learning to parse breast pathology reports
    Yala A; Barzilay R; Salama L
  • Do neural information extraction algorithms generalize across institutions
    Santus E; Li C; Yala A