Background: We examined the comparative performance of structured, diagnostic codes vs. natural language processing (NLP) of unstructured text for screening suicidal behavior among pregnant women in electronic medical records (EMRs). Methods: Women aged 10–64 years with at least one diagnostic code related to pregnancy or delivery (N = 275,843) from Partners HealthCare were included as our “datamart.” Diagnostic codes related to suicidal behavior were applied to the datamart to screen women for suicidal behavior. Among women without any diagnostic codes related to suicidal behavior (n = 273,410), 5880 women were randomly sampled, of whom 1120 had at least one mention of terms related to suicidal behavior in clinical notes. NLP was then used to process clinical notes for the 1120 women. Chart reviews were performed for subsamples of women. Results: Using diagnostic codes, 196 pregnant women were screened positive for suicidal behavior, among whom 149 (76%) had confirmed suicidal behavior by chart review. Using NLP among those without diagnostic codes, 486 pregnant women were screened positive for suicidal behavior, among whom 146 (30%) had confirmed suicidal behavior by chart review. Conclusions: The use of NLP substantially improves the sensitivity of screening suicidal behavior in EMRs. However, the prevalence of confirmed suicidal behavior was lower among women who did not have diagnostic codes for suicidal behavior but screened positive by NLP. NLP should be used together with diagnostic codes for future EMR- based phenotyping studies for suicidal behavior. Keywords: Natural language processing, Electronic medical records, Pregnancy, Suicidal behavior, Screening, Diagnostic codes, Clinical notes Background risk for suicide and, therefore, can help to prevent mater- Suicide, a devastating event, is one of the leading cause of nal mortality [3–5]. However, low-cost, highly scalable maternal deaths during pregnancy and the peripartum methods to identify suicidal behavior are lacking. To period [1, 2]. Early detection of pregnant women with date, studies have primarily relied on the International nonfatal suicidal thoughts and behavior (hereafter referred Classification of Diseases (ICD) billing codes using admin- to as suicidal behavior) presents an important opportunity istrative or claims data to identify instances of suicidal be- for directing suicide prevention efforts to those at high havior [5–9]. Suicidal behavior is often “under-coded” with only a small proportion of suicidal cases being de- tected by the ICD codes among all suicidal cases (i.e., low * Correspondence: firstname.lastname@example.org Department of Epidemiology, Harvard T.H. Chan School of Public Health, sensitivity) [10–13]. For example, a systematic review  Boston, MA 02115, USA reported that the sensitivity of one widely used ICD-9 Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 2 of 11 code category, suicide and self-inflicted injury (E950– Methods E959), ranged from 13.8 to 65%. Using a large primary Data source and study population care database from the United Kingdom (UK), Thomas et We extracted data from the Partners HealthCare System al.  reported that the use of diagnostic codes to detect Research Patient Data Registry (RPDR). The RPDR is a suicidal cases missed approximately three-quarters of the centralized clinical data warehouse for 4.6 million patients cases. The reported low sensitivity of billing codes for from two large academic medical centers (Massachusetts identifying suicidal behavior implies that a sizable portion General Hospital [MGH] and Brigham and Women’s of suicidal cases may be missed when case-finding relies Hospital [BWH]), as well as community and specialty on ICD codes alone. Therefore, expanded data collection hospitals in the Boston area. The RPDR includes structured methods for suicidal behavior are urgently needed to pro- and unstructured EMR information, including socio- vide a foundation for prevention efforts [9, 14]. demographic data, vital signs, laboratory and test results, The increasing utilization of electronic medical records problem list entries, prescribed medications, billing codes, (EMRs) has provided unprecedented opportunities for and clinical notes for healthcare services provided within identifying pregnant women with suicidal behavior. EMRs the system . The Institutional Review Board of Part- contain a ready repository of clinical and phenotypic ners HealthCare (Protocol Number: 2016P000775/BWH) information consisting of structured and unstructured and Harvard T.H. Chan School of Public Health (Protocol data that can enable low-cost population-based studies Number: IRB16–0899) approved all aspects of this study. [15, 16]. Structured data are entered by “clicking” on We initially identified women aged 10–64 years with at choices of lists, forms, or templates, including demographic least one diagnostic code related to pregnancy or delivery data, laboratory test results, and diagnostic billing codes (International Classification of Diseases-10 [ICD-10]: such as the aforementioned ICD codes [16–18]. Unstruc- Z3A.*, O0.*- O9.*; ICD-9: 640.*- 679.*, V22.*, V23.*, V24.*, tured data—clinical data extracted from free-text such V27.*, V28.*; Diagnosis-Related Group [DRG]: 370–384) as physicians’ notes and radiology reports—offers a in the EMRs from January 1, 1996 to March 31, 2016, to- valuable resource for defining clinical phenotypes taling 275,843 women (hereafter referred to as “datamart”) [19–22]. The automated examination of a large vol- included in the datamart (Fig. 1). ume of clinical notes requires the use of natural language processing (NLP) , a field of computational linguistics Suicidal behavior screened positive by diagnostic codes that allows computers to extract relevant information We first screened for suicidal behavior based on diag- from unstructured human language . NLP has been nostic codes including the ICD codes and the Longitu- used successfully to identify patient cohorts for different dinal Medical Record (LMR) codes. The LMR codes phenotypes including treatment resistant depression, bi- were assigned to problem list conditions in the ambula- polar disorder, cerebral aneurysms, rheumatoid arthritis, tory EMR system used across Partners HealthCare Sys- Crohn’s disease, ulcerative colitis, and diabetes [15, 23– tem. (Additional file 1: Table S1). In addition to the 32]. However, very few studies have used NLP to explicit diagnostic codes for suicidal ideation (e.g., identify suicidal behavior in EMRs [10, 33, 34], and ICD-9 V62.84) and suicide attempt (e.g., ICD-9 E95*), no study has reported any classification algorithm we also included additional sets of ICD code categories that is highly predictive of suicidal behavior. (poisoning by analgesics, antipyretics, and antirheumatics; Because of the low prevalence of suicidal behavior poisoning by sedatives and hypnotics; and poisoning by [4, 35], developing a phenotyping algorithm using the psychotropic agents) with positive predictive value ≥0.8 full EMR population would likely result in low positive for suicidal behavior, based on a previous study . predictive values (PPV) . To address this, we first Among the 275,843 women with at least one diagnostic screened for patients with medical record information code related to pregnancy or delivery, 2433 women had at (structured or unstructured) suggestive of suicidal behav- least one diagnostic code related to suicidal behavior, of ior and excluding those with no evidence of suicidal be- which 196 had a diagnostic code that occurred during havior . The patients who screened positive for pregnancy, or within 42 days after abortion or delivery suicidal behavior would serve as a highly sensitive data- . These 196 women, who screened positive for sui- mart and then can be used to develop highly predictive cidal behavior based on diagnostic codes, hereafter classification algorithms for suicidal behavior. Here, using will be referred to as the “diagnostic codes group” (Fig. 1). EMRs from a large healthcare system (Partners Health- Care), we demonstrate that using diagnostic codes to- gether with NLP can more effectively screen for pregnant Suicidal behavior screened positive by NLP-processed women with a higher potential of suicidal behavior. We clinical notes also compare the characteristics of patients identified by Among the 273,410 women without any diagnostic these two methods. codes related to suicidal behavior, we randomly sampled Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 3 of 11 Fig. 1 Screening for suicidal behavior using diagnostic codes vs. NLP among pregnant women. Abbreviations: Natural Language Processing (NLP). Comparative health: the total number of facts which included diagnostic codes for diseases, medications, and specific test results from hospital visits for each patient; it can be used as a proxy for healthcare utilization a subset of women (N= 5880) who were matched for the Systemized Nomenclature of Medicine-Clinical Terms age (10-year intervals), race, and comparative health (SNOMED-CT) , and assigns each term a UMLS con- with the diagnostic codes group using a 1:30 matching cept unique identifier (CUI). cTAKES also extracts quali- ratio. The reason we chose the 1:30 ratio for subsequent fying attributes (including negation, temporality, and NLP was twofold: (1) to provide a sample size that was subject status) associated with each CUI. As determined large enough for a general view of distributions of CUIs, by cTAKES negation module , each CUI can be either and (2) to minimize the NLP processing time. Compara- affirmed (e.g., “patient reports feeling suicidal”)or negated tive health, a proxy for healthcare utilization, was defined (e.g., “suicidal behavior: none”). Affirmed CUIs were con- as the total number of observations in the medical records sidered as relevant for this analysis. cTAKES has a tempor- which included diagnostic codes for diseases, medications, ality module, DocTimeRel (Document Time Relation), to and specific test results from hospital visits for each pa- discover the temporal relation between a term and the tient . To comply with the IRB, Partners HealthCare document creation time . The values for DocTimeRel employees (N= 598) were excluded, leaving 5282 women include “before” (e.g., “patient attempted suicide when she in the matched set. We then searched women’s clinical was 14”), “after” (e.g., “She would not consider suicide an notes and identified 1120 (21.2%) women with at least one option if symptoms were to arise”), “overlap” (e.g., “patient mention of the terms related to suicidal behavior  states that she wants to kill herself”), and “before/overlap” (Additional file 1: Table S2) during pregnancy or within (terms that started before document creation time and the 42 days after abortion or delivery . continue to the present [e.g., “patient endorses passive sui- We further processed the clinical notes of the 1120 cidal ideation since the birth of her baby”]). Terms tagged women using the clinical Text Analysis and Knowledge Ex- as “overlap” or “before/overlap” were considered as tem- traction System (cTAKES 3.2.3, http://ctakes.apache.org/) porally relevant for this analysis. The Subject module indi- . Based on the Unstructured Information Management cates whether the patient or someone else (e.g., “mom Architecture (UIMA), cTAKES is a comprehensive clinical attempted suicide”) experiences the event. The values for NLP tool that processes clinical notes and identifies terms. the Subject module include “patient,”“family member,” cTAKES maps the terms to a subset of the Unified “other,” and “null.”  The terms tagged as “patient” Medical Language System (UMLS) Metathesaurus , were considered as subject relevant for this analysis. Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 4 of 11 We created an expert-defined list of CUIs considered behavior by the diagnostic codes versus NLP during en- relevant to suicidal behavior (Additional file 1: Table S3). counters with suicidal behavior. We examined the distri- We included the distributions of attributes of the CUIs butions of demographic characteristics between pregnant relevant to suicidal behavior in Additional file 1: Table S4. women screened positive for suicidal behavior by the diag- To compensate for errors introduced by the NLP sys- nostic codes versus NLP using the Chi-square test for cat- tem, we calculated the proportion of affirmed, tempor- egorical variables and Student’s t-test for continuous ally relevant, and subject relevant CUIs related to variables. We reported the proportions of women who re- suicidal behavior among all CUIs related to suicidal be- ceived diagnoses of psychiatric comorbidities at least once havior for each woman and selected women with pro- during or before the most recent encounter with suicidal portions that were greater than or equal to 0.25. This behavior. Psychiatric comorbidities were defined using the threshold was determined empirically with an aim to de- ICD codes in Additional file 1: Table S5. All analyses were crease false positives, while maintaining relatively low done using R . false negatives. From the NLP-processed clinical notes, we identified 486 pregnant women (hereafter referred to Results as the “NLP group”) with CUIs related to suicidal behav- We identified 682 pregnant women who screened positive ior. Of note, the NLP group was screened positive by for suicidal behavior, of whom 196 (28.73%) were identi- both term mentions related to suicidal behavior and fied by diagnostic codes and 486 (71.26%) were identified cTAKES. The remainder (N = 634) who had at least one by NLP. Based on manual chart review, the prevalence of mention of the terms related to suicidal behavior during confirmed suicidal behavior in women screened positive pregnancy or within the 42 days after abortion or deliv- (PPV) by the diagnostic codes and by NLP in women ery, but were not screened positive by the NLP are re- without the diagnostic codes were 76.00 and 30.00%, re- ferred to as the “NLP not relevant group.” spectively. The estimated number of confirmed suicidal behavior among the screen positive groups by the diag- Reference group nostic codes and NLP would be 149 and 146, respectively. We randomly sampled a subset of women aged 10–64 The prevalence of confirmed suicidal behavior was 1.00% years with at least one diagnostic code related to preg- among the NLP not relevant group. The prevalence of nancy or delivery as the reference group. The reference confirmed suicidal behavior was 0.00% among women group was matched with comparative health  for the who had neither diagnostic codes nor term mentions re- diagnostic codes group using a 1:100 matching ratio. lated to suicidal behavior. The approximate estimated Since we did not need to process the clinical notes for prevalence of suicidal behavior in the reference group reference group, we included a relatively larger sample would be 2.76% (486 × 0.3/5282). size. After excluding Partners HealthCare employees, The demographic characteristics of women who screened 17,183 women were included in the reference group. positive for suicidal behavior by the diagnostic codes and NLP, respectively, are presented in Table 1.Comparedwith Chart review to obtain estimates for prevalence of the NLP group, the diagnostic codes group was less likely confirmed suicidal behavior to be Hispanic (33.33% vs. 28.57%), be married/common-- After the screening process, one of the authors (QYZ) law married/partnered (29.63% vs. 21.43%), report religious manually reviewed the clinical notes for random samples affiliation as Christian (45.47% vs. 38.27%), and have private of (1) 50 women from the diagnostic codes group (N = insurance (44.65% vs. 32.14%); these women were more 196); (2) 100 women from the NLP group (N = 486); (3) likely to be Black or African American (16.46% vs. 20.92%), 100 women from the NLP not relevant group (N = 634); be single (65.02% vs. 71.43%), and be insured by Medicaid and (4) 100 women who had neither diagnostic codes nor (43.21% vs. 49.49%) and Medicare (6.17% vs. 9.18%). term mentions related to suicidal behavior (N = 4162). Table 2 shows provider characteristics for participants’ Based on the Columbia Classification Algorithm of Sui- encounters (inpatient or outpatient visits) with suicidal be- cide Assessment (C-CASA), the reviewer assigned each havior. For encounters with suicidal behavior, more than woman a classification of either “with” or “without suicidal two-thirds of women in the diagnostic codes group behavior” . Women who had (1) completed suicide, (69.39%) visited the Emergency Department, whereas only (2) suicide attempt, (3) preparatory acts toward imminent 17.49% of women in the NLP group visited the Emergency suicidal behavior, or (4) suicidal ideation were considered Department. The proportions of women screened posi- as “with” suicidal behavior. tive for suicidal behavior treated in an inpatient setting was higher among those in the diagnostic codes group Statistical analysis (39.29%), as compared with those in the NLP group We compared the demographic and provider characteris- (19.55%). tics of pregnant women screened positive for suicidal Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 5 of 11 Table 1 Demographic characteristics of pregnant women screened positive for suicidal behavior by diagnostic codes vs. NLP a b c Characteristics Diagnostic codes (N = 196) NLP (N = 486) P-values Reference group (N = 17,183) n% n % n % d e Age at the most recent pregnancy with suicidal behavior 26.8 (6.9) 26.4 (6.2) 0.46 35.7 (8.3) Age at the most recent pregnancy with suicidal behavior 0.14 < 16 4 2.04 2 0.41 38 0.22 [16, 18) 8 4.08 27 5.56 65 0.38 [18, 20) 16 8.16 53 10.91 205 1.19 [20, 35) 141 71.94 352 72.43 7594 44.19 ≥ 35 27 13.78 52 10.70 9281 54.01 Language 0.51 English 168 85.71 407 83.74 14,844 86.39 Spanish 21 10.71 66 13.58 1537 8.94 Other 7 3.57 13 2.67 802 4.67 Race/ethnicity 0.20 Asian 7 3.57 8 1.65 1036 6.03 Black or African American 41 20.92 80 16.46 1837 10.69 Hispanic 56 28.57 162 33.33 2677 15.58 White 87 44.39 211 43.42 10,413 60.60 Other/Not recorded 5 2.55 25 5.14 1220 7.10 Religion 0.34 Christian 75 38.27 221 45.47 5904 34.36 Catholic 63 32.14 162 33.33 6627 38.57 Islamic 5 2.55 8 1.65 451 2.62 Jewish 1 0.51 6 1.23 676 3.93 No preference/None 27 13.78 49 10.08 1275 7.42 Other/Unknown/Not recorded 25 12.76 40 8.23 2250 13.09 Marital status 0.15 Married/Partner/Common law 42 21.43 144 29.63 10,816 62.95 Single 140 71.43 316 65.02 4675 27.21 Separated/Divorced/Widowed 9 4.59 15 3.09 1000 5.82 Other/Unknown 5 2.55 11 2.26 692 4.03 Vital status 0.04 f f Deceased with date of death 2 1.02 6 1.23 168 0.98 Deceased with date of death unknown 4 2.04 1 0.21 35 0.20 Not reported as deceased 190 96.94 479 98.56 16,980 98.82 Veteran 0.12 No 169 86.22 421 86.63 14,810 86.19 Yes 3 1.53 1 0.21 75 0.44 Unknown 24 12.24 64 13.17 2298 13.37 Insurance 0.03 Medicaid 97 49.49 210 43.21 3097 18.02 Medicare 18 9.18 30 6.17 531 3.09 Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 6 of 11 Table 1 Demographic characteristics of pregnant women screened positive for suicidal behavior by diagnostic codes vs. NLP (Continued) a b c Characteristics Diagnostic codes (N = 196) NLP (N = 486) P-values Reference group (N = 17,183) Private Insurance 63 32.14 217 44.65 12,415 72.25 Self-pay 8 4.08 11 2.26 488 2.84 Other 10 5.10 18 3.70 652 3.79 Abbreviations: Natural language processing (NLP) Randomly sampled from women aged 10–64 years with at least one diagnostic code related to pregnancy or delivery, matching on age, race, comparative health with women screened positive for suicidal behavior by diagnostic codes using a 1:30 matching ratio For continuous variables, P-value was calculated using the Student’s t test; for categorical variables, P-value was calculated using the Chi-square test. Randomly sampled from women aged 10–64 years with at least one diagnostic code related to pregnancy or delivery, matching on comparative health with women screened positive for suicidal behavior by diagnostic codes using a 1:100 matching ratio Mean (Standard deviation) Age at most recent date with diagnostic codes related to pregnancy or delivery None of the deaths occurred within 183 days after suicidal behavior Psychiatric comorbidities were common among women suicidal behavior among pregnant women from a large with suicidal behavior (Table 3). Women screened positive EMR system. More than two-thirds of potential sui- for suicidal behavior by the diagnostic codes had higher psy- cidal behavior and nearly half of confirmed suicidal be- chiatric comorbidities including depression, schizophrenia, havior would have been missed if screening had relied bipolar disorder, post-traumatic stress disorder (PTSD), and solely on ICD codes. However, we observed that the substance abuse. The distribution of care providers accord- PPV of NLP, the probability that a suicidal case identi- ing to clinical specialties (Department of Psychiatry/Mental fied by NLP was truly suicidal, was lower (30.00%) as Health/Behavioral Health and Emergency Department) were compared to the diagnostic codes (76.00%). We found similar across psychiatric comorbidities (Table 3). that womeninthe diagnostic codes group hadmore risk factors for suicidal behavior , including low Discussion socioeconomic status, being single, and psychiatric co- We demonstrated that the use of NLP along with term morbidities as compared with those women in the search substantially improved the sensitivity of screening NLP diagnostic group. Table 2 Provider characteristics at encounters with suicidal behavior of pregnant women screened positive for suicidal behavior by diagnostic codes vs. NLP Characteristics Diagnostic codes (N = 196) NLP (N = 486) Reference group (N = 17,183) n% n % n % Hospitals (ever) Massachusetts General Hospital 105 53.57 249 51.23 12,678 73.78 Brigham and Women’s Hospital 95 48.47 250 51.44 12,800 74.49 Faulkner Hospital 0 0.00 1 0.21 4263 24.81 North Shore Medical Center 0 0.00 11 2.26 2659 15.47 Newton-Wellesley Hospital 0 0.00 4 0.82 5643 32.84 Spaulding Rehabilitation Hospital 0 0.00 2 0.41 1199 6.98 McLean Hospital 0 0.00 0 0.00 283 1.65 Clinics (ever) Emergency 136 69.39 85 17.49 9749 56.74 Psychiatry/Mental health/Behavioral health 7 3.57 221 45.47 2605 15.16 Obstetrics and Gynecology 12 6.12 18 3.70 11,386 66.26 Pediatrics 4 2.04 125 25.72 1253 7.29 Inpatient/outpatient (ever) Inpatient 77 39.29 95 19.55 14,116 82.15 Outpatient 157 80.10 407 83.74 17,055 99.26 Not recorded 0 0.00 46 9.47 9762 56.81 Abbreviations: Natural language processing (NLP) Provider characteristics during lifetime (ever) Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 7 of 11 Table 3 Psychiatric comorbidities of pregnant women screened positive for suicidal behavior by diagnostic codes vs. NLP Psychiatric Diagnostic codes (N = 196) NLP (N = 486) Reference group (N = 17,183) Comorbidities n% n % n % Psychiatric Comorbidities Depression 171 87.24 353 72.63 5150 29.97 Schizophrenia 10 5.10 10 2.06 95 0.55 Bipolar 45 22.96 40 8.23 489 2.85 PTSD 58 29.59 76 15.64 590 3.43 Substance abuse 102 52.04 173 35.60 2332 13.57 Anxiety 100 51.02 259 53.29 5184 30.17 Psychiatric comorbidities at encounters to Department Psychiatry/Mental Health/Behavioral Health (ever) Depression 93 47.45 180 37.04 1536 8.94 Schizophrenia 7 3.57 4 0.82 43 0.25 Bipolar 18 9.18 26 5.35 234 1.36 PTSD 29 14.80 44 9.05 312 1.82 Substance abuse 43 21.94 36 7.41 177 1.03 Anxiety 30 15.31 113 23.25 1058 6.16 Psychiatric comorbidities at encounters to Emergency Department (Ever) Depression 128 65.31 80 16.46 608 3.54 Schizophrenia 6 3.06 2 0.41 22 0.13 Bipolar 25 12.76 8 1.65 76 0.44 PTSD 28 14.29 9 1.85 54 0.31 Substance abuse 74 37.76 57 11.73 652 3.79 Anxiety 51 26.02 42 8.64 660 3.84 Abbreviations: Natural language processing (NLP), post-traumatic stress disorder (PTSD) Prior studies have attempted to identify patients with history. The intersection of both ICD diagnostic codes suicidal behavior in unstructured clinical notes. Using the and the NLP algorithm identified 260 potential cases. The UK Clinical Practice Research Datalink, Thomas et al. positive predictive values for the ICD diagnostic codes found that searching for terms related to suicide in gen- and the NLP algorithm were similar (55% for ICD and eral practice consultation records identified 10.7% of the 60% for NLP). Despite the different NLP tools used across suicidal cases that were missed by ICD diagnostic codes EMR systems, these results consistently suggested that . Anderson et al.  processed the History of Present suicidal behavior was often documented in clinical notes Illness notes of 15,761 patients with at least one diagnostic without being assigned any diagnostic codes that were de- code of depression in primary care clinical organizations. signed for billing purposes. Suicidal behavior is a complex A rule-based NLP system was developed to search for phenotype coupled with many psychosocial problems, positive mention or negation of suicidal behavior using a where clinical notes are often used to capture the com- list of terms related to suicidal behavior. The proportion plexity and diagnostic uncertainty [47, 48]. Incorporating of patients with corresponding ICD diagnostic codes indi- information from unstructured clinical notes through cating suicidal ideation and suicide attempt in the notes NLP in our study, we were able to screen a significant were 3% and 19%, respectively. Haerian et al. used an number of patients with potential suicidal behavior that NLP tool, the Medical Language Extraction and Encoding would otherwise not be found using structured data alone. System (MedLEE), to identify suicidal behavior in the However, the PPV of NLP used in the current study was EMRs for pediatric and adult inpatients. Of note, they lower than that of the diagnostic codes. Nonetheless, we used a list of CUIs with a specific focus on suicidal behav- identified a comparable number of suicidal cases (149 for ior by drug overdose, which was different from the CUI diagnostic codes vs. 146 for NLP) when using only a sub- list we used in our study. In their study, 469 potential sample of women (5880 out of 273,410) without any diag- cases were identified by the ICD diagnostic codes, and nostic codes related to suicidal behavior for NLP. Despite 4087 were identified by the NLP algorithm after filtering the low PPV of NLP, considering the large number of out CUIs that were negated or associated with family pregnant women without diagnostic codes related to Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 8 of 11 suicidal behavior (N = 273,410) and the fact that suicidal conditions that temporally are neither recent nor histor- behavior was often documented in clinical notes, we ical (e.g., “If she has significant side effects from it such as maintain that NLP procedures may be used to identify lethargy/depression/irritability/suicidal thought, we will more suicidal cases. Therefore, for future studies using change it to LTG.”). EMR-based phenotyping for suicidal behavior, an optimal We found that women in the diagnostic codes group approach to increase screening sensitivity may best involve had different characteristics as compared to women in combining the application of NLP procedures with the the NLP group. On the one hand, these differences diagnostic codes. could be due to the lower prevalence of confirmed sui- Only 30% of the women who screened positive for sui- cidal behavior in the NLP group. Therefore, developing cidal behavior by NLP were confirmed to be suicidal by highly predictive classification algorithms is needed for chart review (PPV = 0.30). A large proportion of women the NLP group. On the other hand, the differences be- who were not suicidal were screened positive for suicidal tween women screened positive for suicidal behavior by behavior by NLP. Similar to one previous study , the the diagnostic codes and NLP suggest that the two majority of the false positives came from the incorrect groups may differ with respect to the degree of suicide qualifying attributes based on our error analysis by manual intent, methods used, and subsequent clinical manage- review of the clinical notes from 100 women in the NLP ment. Because a larger proportion of women screened group, in particular, negation associated with CUIs. Neg- positive by the diagnostic codes received inpatient care ation is a well-known challenge for processing unstruc- and were seen in the Emergency Department, they were tured clinical notes . One study showed that likely to present as more severe cases of suicidal behav- approximately half of the conditions indexed in dictated ior with high suicide intent , requiring hospital ad- reports were negated [50, 51]. For suicidal behavior, clini- mission and immediate care. In addition, the diagnostic cians are likely to document both the presence and ab- codes for suicidal ideation (ICD-9: V62.82) were not sence of suicidal behavior . In the Partners HealthCare used until October 2005 when the codes were intro- EMRs, we observed a major negation structure for suicidal duced. Even after the codes became available, one study behavior: terms related to suicidal behavior were followed showed that suicidal ideation was less likely to be coded by a colon and a negation word without any sentence than suicide attempt . These two factors (i.e., source punctuation (e.g., “suicidal behavior: none,”“suicidal be- of inpatient care and timing of availability of diagnostic havior: none reported,” and “suicidal behavior: denied”) codes) might have contributed to a disproportionate rep- (Additional file 1: Table S6). However, the standard resentation of more severe cases of suicidal behavior in cTAKES negation module NegEx [40, 52], a regular ex- the diagnostic codes group. In this scenario, women pression pattern matching algorithm that searches for pre- screened positive by the diagnostic codes may be a more defined negation words around terms  was initially relevant cohort for assessing patients at high risk for trained using the Intensive Care Unit discharge summar- completed suicide , whereas women screened posi- ies , and is not able to recognize such negation struc- tive by NLP may be more relevant for investigating early ture . Consequently, a considerable number of suicidal identification of high-risk groups and suicide prevention behavior terms that were negated were incorrectly identi- interventions. Another possibility for the observed dif- fied as “affirmed.” Further enhancement of the negation ferences in characteristics, especially for psychiatric co- algorithm with training data pertaining specifically to sui- morbidities, between the diagnostic codes group and the cidal behavior is required to decrease the false positives NLP group could be due to differential bias in coding: [49, 55]. Other common reasons leading to cTAKES mis- women with more risk factors were more likely to be coding women without suicidal behavior as suicidal (Add- coded for suicidal behavior. itional file 1: Table S6) included (1) incorrect recognition There are several limitations of this study. First, the of “before” as “overlap” by the DocTimeRel module (e.g., prevalence of confirmed suicidal behavior among women DocTimeRel module treated history of suicidal behavior as screened positive by NLP was only 30%. However, given current suicidal behavior: “Suicide attempt/gesture: his- the purpose of our study, which was to screen pregnant tory of, hospitalized inpatient psych unit for suicide at- women with a higher potential of suicidal behavior and tempt in 1996”); (2) incorrect recognition of “family to develop a highly sensitive datamart for suicidal behav- member” as “patient” by the Subject module (e.g., Subject ior, this low PPV might be tolerated. Nevertheless, using module treated the suicidal behavior of patient’sfatheras this highly sensitive datamart for suicidal behavior, fu- patient’s: “Pt also identifies strongly with father, who was ture development of accurate classification algorithms often aggressive toward others and threatened suicide”); using different machine learning techniques [58, 59]is (3) failure to identify section titles (e.g., “Suicidal Behavior clearly needed to identify true cases of suicidal behavior. Hx of Suicidal Behavior:”) that do not describe the behav- Second, given the small sample size of women screened ior of patients; and (4) failure to handle hypothetical positive for suicidal behavior by the diagnostic codes, we Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 9 of 11 did not further classify patients according to subtypes of Additional file suicidal behavior such as suicidal ideation and suicide at- Additional file 1: Table S1. International Classification of Disease (ICD) tempt. Third, given that a woman was considered as codes and other diagnostic codes used to screen suicidal behavior. Table screened positive for suicidal behavior only if she was S2. Terms used to screen suicidal behavior in clinical notes. Table S3. screened positive by both term mention and NLP by Concept Unique Identifiers (CUIs) related to suicidal behavior. Table S4. Distributions of attributes of the Concept Unique Identifiers (CUIs) related cTAKES, it is possible that we might miss some women to suicidal behavior among 1120 women. Table S5. International who did not pass the screening by term mentions related Classification of Disease (ICD) codes used to define psychiatric to suicidal behavior but would have been considered as comorbidities. Table S6. Error analysis of false positive results from cTAKES to screen for suicidal behavior. (DOCX 36 kb) screened positive by cTAKES. Fourth, we used 20 years of data from a single urban-regional EMR system that did not include patient visits outside this geographical area, Abbreviations BWH: Brigham and women’s hospital; C-CASA: Columbia classification time period, or network of hospitals. The generalizability algorithm of suicide assessment; cTAKES: clinical Text Analysis and of our results to patients in other healthcare systems may Knowledge Extraction System; CUI: Concept unique identifier; vary depending on the informatics infrastructure and local DRG: Diagnosis-related group; EMRs: Electronic medical records; ICD: International classification of diseases; LMR: Longitudinal medical record; documentation practices . Fifth, we focused on extract- MedLEE: Medical language extraction and encoding system; ing facts expressed directly in the clinical notes (i.e., terms MGH: Massachusetts general hospital; NLP: Natural language processing; of suicidal behavior) using NLP. However, beyond extract- PPV: Positive predictive values; RPDR: Research patient data registry; SNOMED-CT: Systemized nomenclature of medicine-clinical terms; ing these basic facts, further research in studying other lin- UMLS: Unified medical language system guistic features, such as sentiment expressed in clinical notes (e.g., positive and negative emotions), and capturing Acknowledgements The authors are very grateful for the help of Leslie Howes at Harvard T.H. the meaning of texts (e.g., word embedding [60–62]), may Chan School of Public Health, and the Harvard Catalyst Leadership Team also be beneficial in identifying suicidal patients [63–67]. during the planning and development of this research. The authors thank the Enterprise Research Infrastructure & Services at Partners HealthCare for the provision of computing resources. The authors also thank Laurie Bogosian and Stacey Duey of the Research Patient Data Repository at Conclusion Partners HealthCare for the in-depth support. The authors thank Kathy Bren- Our results illuminated the advantage of using NLP ner for the help with editing this manuscript. This research was done as along with term search in EMRs to screen pregnant partial fulfillment of the requirements of a Doctor of Science degree by one of the authors (QYZ) in the Department of Epidemiology, Harvard women for a complex, rare psychiatric phenotype. NLP T.H. Chan School of Public Health, Boston, MA, USA. One of the authors substantially improved the sensitivity of screening for (QYZ) expresses appreciation to Dr. Michael Napolitano for his non- suicidal behavior in an obstetric population. We cap- expert comments, and constant support and encouragement in complet- ing this manuscript. tured a group of pregnant women with potential suicidal behavior otherwise not reflected in the structured data. Funding We also highlight the challenges of using NLP in This research was supported by awards from the National Institutes of Health screening pregnant women for suicidal behavior. Of (the National Institute on Minority Health and Health Disparities: T37- MD001449; and the National Center for Research Resources (NCRR), the Na- note, NLP had lower PPV as compared with diagnos- tional Center for Advancing Translational Sciences (NCATS): 8UL1TR 000170– tic codes. Improvement in the cTAKES modules, 09). The NIH had no further role in study design; in the collection, analysis especially the negation module, may help to increase and interpretation of data; in the writing of the manuscript; and in the deci- sion to submit the paper for publication. the PPV. For future studies using EMR-based pheno- typing for suicidal behavior, an optimal approach Availability of data and materials may include combining NLP procedures with the The data that support the findings of this study are available from Partners HealthCare but restrictions apply to the availability of these data, which were diagnostic codes. used under license for the current study, and so are not publicly available. Our approach is the first to examine the large-scale Data are however available from the authors upon reasonable request and use of NLP in suicidal behavior among pregnant women. with permission of Partners HealthCare. The current study in our population of pregnant women Authors’ contributions was particularly challenging given the rarity of suicidal QYZ (email@example.com), EWK (firstname.lastname@example.org), BG behavior, the stigma attached, the complexity of pheno- (email@example.com), PA (Paul_Avillach@hms.harvard.edu), JWS typic assessment, and the historical misconception of the (firstname.lastname@example.org), TC (email@example.com), and MAW (firstname.lastname@example.org) conceived and designed the study. QYZ protective role of pregnancy in suicidal behavior [3, 68]. processed and analyzed the data with support from SF Because pregnancy is a time when women have frequent (Sean.Finan@childrens.harvard.edu). All authors interpreted the data. QYZ interactions with the healthcare system, EMR-based iden- wrote the manuscript with input from all authors. All authors read and approved the final manuscript. tification of pregnant women with suicidal behavior may be useful for future genetic, epidemiological, and clinical Ethics approval and consent to participate studies, presenting a valuable opportunity for healthcare The Institutional Review Board (IRB) of Partners HealthCare (Protocol Number: providers to intervene promptly [5, 69]. 2016P000775/BWH) and Harvard T.H. Chan School of Public Health (Protocol Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 10 of 11 Number: IRB16–0899) approved all aspects of this study. The IRB granted a 18. de Lusignan S, van Weel C. The use of routinely collected computer data for waiver of consent/authorization. research in primary care: opportunities and challenges. Fam Pract. 2006;23:253–63. 19. Bates DW, Evans RS, Murff H, Stetson PD, Pizziferri L, Hripcsak G. Detecting adverse events using information technology. J Am Med Inform Assoc. Competing interests 2003;10:115–28. The authors declare that they have no competing interest. 20. Pakhomov SVS, Shah ND, Van Houten HK, Hanson PL, Smith SA. The role of the electronic medical record in the assessment of health related quality of life. AMIA Annu Symp Proc. 2011;2011:1080–8. Publisher’sNote 21. Fischer LR, Rush WA, Kluznik JC, O’Connor PJ, Hanson AM. Abstract C-C1-06: Springer Nature remains neutral with regard to jurisdictional claims in Identifying Depression Among Diabetes Patients Using Natural Language published maps and institutional affiliations. Processing of Office Notes. Clin Med Res. 2008;6:125–6. 22. Jha AK. The promise of electronic records: around the corner or down the road? Author details JAMA. 2011;306:880–1. Department of Epidemiology, Harvard T.H. Chan School of Public Health, 23. Raymond Francis Sarmiento FD. Improving Patient Cohort Identification Boston, MA 02115, USA. Department of Medicine, Division of Rheumatology, Using Natural Language Processing. In: MIT Critical Data, editor. Secondary Immunology and Allergy, Brigham and Women’s Hospital and Harvard Analysis of Electronic Health Records. Berlin, Germany: Springer International Medical School, Boston, MA, USA. Children’s Hospital Informatics Program, Publishing; 2016. p. 405–417. Boston Children’s Hospital, Boston, MA, USA. Department of Biomedical 24. LinC,Karlson EW,Dligach D, Ramirez MP, Miller TA,MoH,et al. Informatics, Harvard Medical School, Boston, MA, USA. Psychiatric and Automatic identification of methotrexate-induced liver toxicity in Neurodevelopmental Genetics Unit, Massachusetts General Hospital, Boston, patients with rheumatoid arthritis from the electronic medical record. J MA, USA. Department of Biostatistics, Harvard T.H. Chan School of Public Am Med Inform Assoc. 2015;22:e151–61. Health, Boston, MA, USA. 25. Castro VM, Dligach D, Finan S, Yu S, Can A, Abd-El-Barr M, et al. Large-scale identification of patients with cerebral aneurysms using natural language Received: 31 October 2017 Accepted: 24 May 2018 processing. Neurology. 2017;88:164–8. 26. Perlis RH, Iosifescu DV, Castro VM, Murphy SN, Gainer VS, Minnier J, et al. Using electronic medical records to enable large-scale studies in References psychiatry: treatment resistant depression as a model. Psychol Med. 1. Oates M. Suicide: the leading cause of maternal death. Br J Psychiatry. 2003; Cambridge Univ Press; 2012;42:41–50. 183:279–81. 27. Castro V, Shen Y, Yu S, Finan S, Pau CT, Gainer V, et al. Identification of 2. Oates M. Perinatal psychiatric disorders: a leading cause of maternal subjects with polycystic ovary syndrome using electronic health records. morbidity and mortality. Br Med Bull. 2003;67:219–29. Reprod Biol Endocrinol. 2015;13:116. 3. Gelaye B, Kajeepeta S, Williams MA. Suicidal ideation in pregnancy: an 28. Castro VM, Apperson WK, Gainer VS, Ananthakrishnan AN, Goodson AP, epidemiologic review. Arch Womens Ment Health. 2016;19:741–51. Wang TD, et al. Evaluation of matched control algorithms in EHR-based 4. Barak-Corren Y, Castro VM, Javitt S, Hoffnagle AG, Dai Y, Perlis RH, et al. phenotyping studies: a case study of inflammatory bowel disease Predicting Suicidal Behavior From Longitudinal Electronic Health Records. comorbidities. J Biomed Inform. 2014;52:105–11. Am J Psychiatry. 2017;174:154–62. 29. Liao KP, Cai T, Gainer V, Goryachev S, Zeng-treitler Q, Raychaudhuri S, et al. 5. Zhong Q-Y, Gelaye B, Miller M, Fricchione GL, Cai T, Johnson PA, et al. Suicidal Electronic medical records for discovery research in rheumatoid arthritis. behavior-related hospitalizations among pregnant women in the USA, 2006- Arthritis Care Res. 2010;62:1120–7. 2012. Arch Womens Ment Health. Springer Vienna; 2016;19:463–472. 30. Liao KP, Ananthakrishnan AN, Kumar V, Xia Z, Cagan A, Gainer VS, et al. 6. Gandhi SG, Gilbert WM, McElvy SS, El Kady D, Danielson B, Xing G, et al. Methods to Develop an Electronic Medical Record Phenotype Algorithm Maternal and neonatal outcomes after attempted suicide. Obstet Gynecol. to Compare the Risk of Coronary Artery Disease across 3 Chronic 2006;107:984–90. Disease Cohorts. PLoS One. 2015;10:e0136651. 7. Patrick AR, Miller M, Barber CW, Wang PS, Canning CF, Schneeweiss S. 31. Ananthakrishnan AN, Cai T, Savova G, Cheng S-C, Chen P, Perez RG, et al. Identification of hospitalizations for intentional self-harm when E-codes are Improving case definition of Crohn’s disease and ulcerative colitis in incompletely recorded. Pharmacoepidemiol Drug Saf. 2010;19:1263–75. electronic medical records using natural language processing: a novel 8. Simon GE, Savarino J. Suicide attempts among patients starting depression informatics approach. Inflamm Bowel Dis. 2013;19:1411–20. treatment with medications or psychotherapy. Am J Psychiatry. 2007;164:1029–34. 32. Carroll RJ, Thompson WK, Eyler AE, Mandelin AM, Cai T, Zink RM, et al. 9. Walkup JT, Townsend L, Crystal S, Olfson M. A systematic review of Portability of an algorithm to identify rheumatoid arthritis in electronic validated methods for identifying suicide or suicidal ideation using health records. J Am Med Inform Assoc. 2012;19:e162–9. administrative or claims data. Pharmacoepidemiol Drug Saf. 2012; 33. Anderson HD, Pace WD, Brandt E, Nielsen RD, Allen RR, Libby AM, et al. 21(Suppl 1):174–82. Monitoring suicidal patients in primary care using electronic health 10. Haerian K, Salmasian H, Friedman C. Methods for identifying suicide or records. J Am Board Fam Med. 2015;28:65–71. suicidal ideation in EHRs. AMIA Annu Symp Proc. 2012;2012:1244–53. 34. Downs JM, Velupillai S, Gkotsis G, Holden R, Kikoler M, Dean H, et al. Detection 11. Colman I, Yiannakoulias N, Schopflocher D, Svenson LW, Rosychuk RJ, Rowe of Suicidality in Adolescents with Autism Spectrum Disorders: Developing a BH, et al. Population-based study of medically treated self-inflicted injuries. Natural Language Processing Approach for Use in Electronic Health CJEM. 2004;6:313–20. Records. Proc AMIA Symp [Internet]. 2017 [cited 2018 Mar 19]; Available 12. Thomas KH, Davies N, Metcalfe C, Windmeijer F, Martin RM, Gunnell D. from: https://kclpure.kcl.ac.uk/portal/en/publications/detection-of- Validation of suicide and self-harm records in the Clinical Practice Research suicidality-in-adolescents-with-autism-spectrum-disorders(2e703fc1- Datalink. Br J Clin Pharmacol. 2013;76:145–57. 2f87-448e-abfc-14e36036c471)/export.html 13. Lu CY, Stewart C, Ahmed AT, Ahmedani BK, Coleman K, Copeland LA, et al. 35. Smoller JW. The use of electronic health records for psychiatric phenotyping How complete are E-codes in commercial plan claims databases? and genomics. Am J Med Genet B Neuropsychiatr Genet [Internet]. 2017; Pharmacoepidemiol Drug Saf. 2014;23:218–20. Available from: https://doi.org/10.1002/ajmg.b.32548 14. US Public Health Service. The Surgeon General's Call to Action to Prevent 36. Liao KP, Cai T, Savova GK, Murphy SN, Karlson EW, Ananthakrishnan Suicide. Washington, DC: US Public Health Service; 1999. AN, et al. Development of phenotype algorithms using electronic 15. Castro VM, Minnier J, Murphy SN, Kohane I, Churchill SE, Gainer V, et al. medical records and incorporating natural language processing. BMJ. Validation of electronic health record phenotyping of bipolar disorder 2015;350:h1885. cases and controls. Am J Psychiatry. 2015;172:363–72. 16. Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better 37. Wang SV, Rogers JR, Jin Y, Bates DW, Fischer MA. Use of electronic research applications and clinical care. Nat Rev Genet. 2012;13:395–405. healthcare records to identify complex patients with atrial fibrillation for 17. Sinnott JA, Dai W, Liao KP, Shaw SY, Ananthakrishnan AN, Gainer VS, et al. targeted intervention. J Am Med Inform Assoc. 2017;24:339–44. Improving the power of genetic association tests with imperfect phenotype 38. World Health Organization. International Statistical Classification of Diseases derived from electronic medical records. Hum Genet. 2014;133:1369–82. and Related Health Problems. In: World Health Organization; 2004. Zhong et al. BMC Medical Informatics and Decision Making (2018) 18:30 Page 11 of 11 39. Murphy SN, Weber G, Mendis M, Gainer V, Chueh HC, Churchill S, et al. 62. Mikolov T, Chen K, Corrado G, Dean J. Efficient Estimation of Word Serving the enterprise and beyond with informatics for integrating biology Representations in Vector Space [Internet]. arXiv [cs.CL]. 2013. Available and the bedside (i2b2). J Am Med Inform Assoc. 2010;17:124–30. from: http://arxiv.org/abs/1301.3781 40. Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. 63. McCoy TH Jr, Castro VM, Roberson AM, Snapper LA, Perlis RH. Improving Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): Prediction of Suicide and Accidental Death After Discharge From General architecture, component evaluation and applications. J Am Med Inform Hospitals With Natural Language Processing. JAMA Psychiatry. 2016;73:1064–71. Assoc. jamia.oxfordjournals.org. 2010;17:507–13. 64. Leonard Westgate C, Shiner B, Thompson P, Watts BV. Evaluation of Veterans’ Suicide Risk With the Use of Linguistic Detection Methods. 41. Bodenreider O. The Unified Medical Language System (UMLS): integrating Psychiatr Serv. 2015;66:1051–6. biomedical terminology. Nucleic Acids Res. 2004;32:D267–70. 65. Roberts A. Language, Structure, and Reuse in the Electronic Health Record. 42. Donnelly KSNOMED-CT. The advanced terminology and coding system for AMA J Ethics. 2017;19:281–8. eHealth. Stud Health Technol Inform. 2006;121:279–90. 66. Pestian J, Nasrallah H, Matykiewicz P, Bennett A, Leenaars A. Suicide Note 43. Chikka VR, Mariyasagayam N, Niwa Y, Karlapalem K. Information Extraction Classification Using Natural Language Processing: A Content Analysis. from Clinical Documents: Towards Disease/Disorder Template Filling. Biomed Inform Insights. 2010;2010:19–28. Experimental IR Meets Multilinguality, Multimodality, and Interaction. Cham: 67. Pestian JP, Grupp-Phelan J, Bretonnel Cohen K, Meyers G, Richey LA, Springer; 2015. p. 389–401. Matykiewicz P, et al. A Controlled Trial Using Natural Language Processing 44. Posner K, Oquendo MA, Gould M, Stanley B, Davies M. Columbia to Examine the Language of Suicidal Adolescents in the Emergency Classification Algorithm of Suicide Assessment (C-CASA): classification of Department. Suicide Life Threat Behav. 2016;46:154–9. suicidal events in the FDA’s pediatric suicidal risk analysis of antidepressants. 68. Appleby L. Suicide during pregnancy and in the first postnatal year. Am J Psychiatry. 2007;164:1035–43. BMJ. 1991;302:137–40. 45. R Core Team. R: A language and environment for statistical computing. 69. Gold KJ, Singh V, Marcus SM, Palladino CL. Mental health, substance use Vienna, Austria: R Foundation for Statistical Computing; 2014. and intimate partner problems among pregnant and postpartum suicide 46. Turecki G, Brent DA. Suicide and suicidal behaviour. Lancet. 2016;387:1227–39. victims in the National Violent Death Reporting System. Gen Hosp 47. Ford E, Carroll JA, Smith HE, Scott D, Cassell JA. Extracting information from Psychiatry. 2012;34:139–45. the text of electronic medical records to improve case detection: a systematic review. J Am Med Inform Assoc. 2016;23:1007–15. 48. Ford E, Campion A, Chamles DA, Habash-Bailey H. “You don’t immediately stick a label on them”: a qualitative study of influences on general practitioners’ recording of anxiety disorders. BMJ Open [Internet]. bmjopen.bmj.com; 2016; Available from: http://bmjopen.bmj.com/content/6/6/e010746.short 49. Wu S, Miller T, Masanz J, Coarr M, Halgrim S, Carrell D, et al. Negation’s not solved: generalizability versus optimizability in clinical natural language processing. PLoS One. 2014;9:e112774. 50. Harkema H, Dowling JN, Thornblade T, Chapman WW. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. J Biomed Inform. 2009;42:839–51. 51. Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. Evaluation of negation phrases in narrative clinical reports. Proc AMIA Symp. 2001:105–9. 52. Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform. 2001;34:301–10. 53. Sohn S, Wu S, Chute CG. Dependency Parser-based Negation Detection in Clinical Narratives. AMIA Jt Summits Transl Sci Proc. 2012;2012:1–8. 54. Garla V, Lo Re V III, Dorey-Stein Z, Kidwai F, Scotch M, Womack J, et al. The Yale cTAKES extensions for document classification: architecture and application. J Am Med Inform Assoc. 2011;18:614–20. 55. Gkotsis G, Velupillai S, Oellrich A, Dean H, Liakata M, Dutta R. Don’t Let Notes Be Misunderstood: A Negation Detection Method for Assessing Risk of Suicide in Mental Health Records. In: Proceedings of the Third Workshop on Computational Lingusitics and Clinical Psychology. Association for Computational Linguistics; 2016. p. 95–105. 56. Crandall C, Fullerton-Gleason L, Aguero R, LaValley J. Subsequent suicide mortality among emergency department patients seen for suicidal behavior. Acad Emerg Med. 2006;13:435–42. 57. Chock MM, Bommersbach TJ, Geske JL, Bostwick JM. Patterns of Health Care Usage in the Year before Suicide: A Population-Based Case-Control Study. Mayo Clin Proc. 2015;90:1475–81. 58. Metzger M-H, Tvardik N, Gicquel Q, Bouvry C, Poulet E, Potinet-Pagliaroli V. Use of emergency department electronic medical records for automated epidemiological surveillance of suicide attempts: a French pilot study. Int J Methods Psychiatr Res [Internet]. Wiley Online. Library. 2017;26 Available from: https://doi.org/10.1002/mpr.1522 59. Walsh CG, Ribeiro JD, Franklin JC. Predicting Risk of Suicide Attempts Over Time Through Machine Learning. Clin Psychol Sci. SAGE Publications Inc. 2017;5:457–69. 60. Jagannatha AN, Yu H. Bidirectional RNN for Medical Event Detection in Electronic Health Records. Proc Conf. 2016;2016:473–82. 61. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed Representations of Words and Phrases and their Compositionality. In: Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, editors. Advances in Neural Information Processing Systems 26. Curran Associates, Inc; 2013. p. 3111–9.
BMC Medical Informatics and Decision Making
– Springer Journals
Published: May 29, 2018