Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You and Your Team.

Learn More →

Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival

Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Key Points Question Can deep learning IMPORTANCE There is a lack of studies exploring the performance of a deep learning survival neural architecture be applied for individual network in non–small cell lung cancer (NSCLC). prognosis evaluation and treatment recommendation? OBJECTIVES To compare the performances of DeepSurv, a deep learning survival neural network Findings In this cohort study of 17 322 with a tumor, node, and metastasis staging system in the prediction of survival and test the reliability patients with non–small cell lung cancer. of individual treatment recommendations provided by the deep learning survival neural network. The performance of a deep learning model was assessed on real-life clinical DESIGN, SETTING, AND PARTICIPANTS In this population-based cohort study, a deep learning– data sets. The ability of a deep learning based algorithm was developed and validated using consecutive cases of newly diagnosed stages I to model to learn complex associations IV NSCLC between January 2010 and December 2015 in a Surveillance, Epidemiology, and End between an individual’s characteristics Results database. A total of 127 features, including patient characteristics, tumor stage, and and the outcome benefits of different treatment strategies, were assessed for analysis. The algorithm was externally validated on an treatments was also elucidated; independent test cohort, comprising 1182 patients with stage I to III NSCLC diagnosed between particularly, a deep learning network January 2009 and December 2013 in Shanghai Pulmonary Hospital. Analysis began January 2018 identified persons with non–small cell and ended June 2019. lung cancer and survival more accurately than tumor, node, metastasis staging. MAIN OUTCOMES AND MEASURES The deep learning survival neural network model was compared with the tumor, node, and metastasis staging system for lung cancer–specific survival. The Meaning Findings suggest that this C statistic was used to assess the performance of models. A user-friendly interface was provided to novel analytical approach may have facilitate the survival predictions and treatment recommendations of the deep learning survival great potential in providing individual neural network model. prognostic information and treatment recommendations in real clinical RESULTS Of 17 322 patients with NSCLC included in the study, 13 361 (77.1%) were white and the scenarios. median (interquartile range) age was 68 (61-74) years. The majority of tumors were stage I disease (10 273 [59.3%]) and adenocarcinoma (11 985 [69.2%]). The median (interquartile range) follow-up Invited Commentary time was 24 (10-43) months. There were 3119 patients who had lung cancer–related death during the follow-up period. The deep learning survival neural network model showed more promising results Video in the prediction of lung cancer–specific survival than the tumor, node, and metastasis stage on the Supplemental content test data set (C statistic = 0.739 vs 0.706). The population who received the recommended treatments had superior survival rates than those who received treatments not recommended Author affiliations and article information are listed at the end of this article. (hazard ratio, 2.99; 95% CI, 2.49-3.59; P < .001), which was verified by propensity score–matched groups. The deep learning survival neural network model visualization was realized by a user-friendly graphic interface. CONCLUSIONS AND RELEVANCE The deep learning survival neural network model shows potential benefits in prognostic evaluation and treatment recommendation with respect to lung (continued) Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 1/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Abstract (continued) cancer–specific survival. This novel analytical approach may provide reliable individual survival information and treatment recommendations. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 Introduction Lung cancer is the most commonly diagnosed cancer in China and the second in the United States, approximately 85% of which is non–small cell lung cancer (NSCLC). The precise stratification of patients with NSCLC into groups according to survival outcomes represents a crucial step in treatment. The staging system in the 8th edition of the American Joint Committee on Cancer classifies patients based on tumor, node, and metastasis (TNM) staging. However, the survival rate 3-5 within the same stage cohort varies widely. It has been found that other independent prognostic factors including age, sex, histology, and treatment choices could significantly contribute to individualized predictions of survival. To improve the precision of lung cancer survival estimations, Cox proportional hazard models have gained popularity as a way of predicting outcomes. For example, the nomogram is a reliable tool that has demonstrated the ability to quantify risk by combining and clarifying significant clinical characteristics for clinical oncology. By drafting a concise chart of an outcome-risk predictive model, a nomogram derives the risk probability of a specific event, such as lung cancer–specific survival (LCSS). In various cancers, nomograms possess the ability to derive more precise risk predictions 8,9 when incorporated with TNM staging. However, these models have several limitations with respect to time-to-event prediction for the clinical management of patients with cancer, including the precise evaluation of overall survival and time to progression. Moreover, these models make linearity assumptions rather than perform nonlinear analyses that reflect real-world clinical 11 12 characteristics. Therefore, there is a need for better solutions that focus on nonlinear variables. Deep learning networks can learn the highly intricate and linear/nonlinear associations between prognostic clinical characteristics and an individual’s risk of death from LCSS. In application, these networks have even shown potential for providing individual recommendations based on the calculated risk. For example, by analyzing clinical data in the Surveillance, Epidemiology, and End Results (SEER) cancer registry, Bergquist et al assembled computerized methods including random forests, lasso regression, and neural networks to achieve 93% accuracy in predicting lung cancer stages. In another study, Corey et al developed a software package (Pythia) based on machine learning models that incorporated a patient’s age, sex, clinical baseline, race/ethnicity, and comorbidity history to determine the risk of postoperative complications or deaths. Matsuo et al also developed a deep learning network model that has demonstrated a higher C statistic than the traditional proportional hazard regression model (C statistic = 0.795 vs 0.784) for progression-free survival analysis. Furthermore, Katzman et al developed a novel deep learning method for survival analysis that uses a deep learning network to integrate Cox proportional hazards, which is referred to as the learning survival neural network (DeepSurv). The authors demonstrated that DeepSurv performed as well as published survival models and could be used to provide treatment recommendations for better survival outcomes. The present study design follows the American Joint Committee on Cancer criteria for model adoption and the transparent report of a deep learning architecture for individual prognosis and treatment recommendation. In this study, we first describe the performance of DeepSurv on real- life clinical data sets. Second, we elucidate how the DeepSurv model can learn complex associations between an individual’s characteristics and the outcome benefits of different treatments. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 2/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Methods Eligibility Criteria and Clinical Information All patients gave informed oral consent prior to data collection. After obtaining institutional review board approval from Shanghai Pulmonary Hospital, we selected patients from the SEER 18 Regs Research Data + Hurricane Katrina Impacted Louisiana Cases, Nov 2017 Sub, which includes clinical records on cancer occurrences in 18 areas of the United States and contains approximately 27.8% of the population. Clinical cases were included if the following criteria were met: pathologically confirmed primary stage I to IV NSCLC (only adenocarcinoma and squamous cell carcinoma) between January 2010 and December 2015 and the presence of 1 malignant primary lesion. From the SEER database (eTable 1 in the Supplement), we collected the baseline information of cases (sex, age, and marriage status), tumor characteristics (location, size, histologic grade, histologic type, TNM stage, SEER code (CS extension, CS mets at dx, regional nodes examined, regional nodes positive, lung– pleural/elastic layer invasion by H and E or elastic stain, lung–separate tumor nodules–ipsilateral lung, 19,20 lung–surgery to primary site]), and treatment details (surgical type). Patients were excluded if any of the included clinical characteristics status were unknown or missing. The outcome of interest in this study included LCSS according to specific codes provided by SEER (defined as the interval from surgery until death as a result of lung cancer). These patients were randomly divided into the training and validation cohort at a ratio of 8 to 2. To validate the DeepSurv model, an external test cohort was provided by the CHINA database. The cohort comprised 1182 patients with stage I to III NSCLC diagnosed between January 2009 and December 2013 in Shanghai Pulmonary Hospital, which are completely distinct from the patients in SEER database. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. Deep Learning Model Design In this study, DeepSurv was used to analyze patient-individual survival outcomes, which is a deep learning algorithm that can predict individual survival risk values (Figure 1). We use deep feed- forward neural network and the Cox proportional hazards model in survival analysis. The DeepSurv model contained a core hierarchical structure with fully connected feed-forward neural networks with a single output node to calculate the survival risks hθ(x ) of patients using the negative log-partial likelihood function. More details about the DeepSurv were described in the eMethods in the Supplement. Using the provided data set, we compared the performances of the TNM staging model and our deep learning model with respect to 2 tasks (LCSS predictions) with 3 different data sets (Figure 1). Data Analysis First, we developed a 6-layer neural network for predicting patient LCSS in the NSCLC training data set (n = 12 912). To validate the prediction performance, we used Harrell C statistic and calibration plots to evaluate the network discrimination and calibration in the NSCLC validation data set (n = 3228) and CHINA data set (n = 1182). Next, we trained a personalized treatment recommendation system using separately developed lobar and sublobar resection risk prediction models with a 3-layer neural network in the lobectomy (n = 10 766) and sublobectomy (n = 1444) training data sets. For each patient in the lobectomy and sublobectomy validation data set (SEER: n = 3064; CHINA: n = 1142), we chose the lower-risk value of the model’s treatment as the recommendation. Finally, we categorized the patients into 2 groups according to the consistency of the treatment received and recommended. For survival analysis, we used the Kaplan-Meier method to analyze LCSS between different groups and the log-rank test to compare survival curves. 6,17 An additional Cox proportional hazard regression model with nonneural network methods was performed following the simple backward-stepwise approach using all the variables included in JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 3/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival the DeepSurv model. It estimated the risk function h(x ) of the event occurring (LCSS) for patient i based on included features x using a linear function: h(Xi) =  x β (β = the coefficient of x ). i i i i Model Visualization We also developed a user-friendly interface to facilitate the survival predictions and treatment recommendations of the DeepSurv model. This interface consists of 3 views: (1) the user input view, (2) the survival prediction view, and (3) the treatment recommendation view. The user input view is designed to help users input all entries regarding patient characteristics using the XML schema constructed based on the features input into DeepSurv models. The user input view allows users to predict the survival probability and obtain a treatment recommendation based on specific patient information by clicking the predict and recommendation buttons, respectively. All SEER codes followed the SEER guideline. Statistical Analysis A 2-sided P value less than .05 was considered to be statistically significant. The Akaike information criterion was calculated to assess the risk of overfitting. The likelihood-based method was applied to the type I censoring design. All statistical analyses were performed with SPSS version 23 (IBM Corporation) software. The C statistic was performed by comparing C package with R statistical software (R Project for Statistical Computing), and the survival curves were plotted using GraphPad Prism 7 (GraphPad Software) software. Codes in our study are available online (https://github.com/thoraciclang/Deep_Lung). Analysis began January 2018 and ended June 2019. Figure 1. Diagram of the Study Procedure Training DeepSurv Test Linear Multilayer perception combination Prediction Survival SEER SEER CHINA h (x) n = 12 912 n = 3228 n = 1182 SEER lobectomy h (x)1 n = 10 766 h (x)1 SEER CHINA Surgery vs n = 3064 n = 1142 h (x)2 SEER sublobar resection h (x)2 n = 1444 Performance Stability Recommendation Visualization was benchmarked was assessed in the was tested by was applied using user-friendly against the TNM model validation cohort survival analysis graphic interface Deep learning networks were trained end to end on 3 data set groups. The training and node, and metastasis (TNM) models, assess their degrees of stability, formulate testing of these networks were all conducted on independent data sets. Four further recommendations, and finally, accomplish model visualization. SEER indicates experiments were conducted on the networks to test their performances against tumor, Surveillance, Epidemiology, and End Results. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 4/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Results Screening Process and Clinicopathology A total of 17 322 patients with NSCLC were included in the study. According to the screening criteria, a total of 16 140 patients diagnosed as having NSCLC from the SEER database were included (eFigure 1A in the Supplement). Table 1 shows the patients’ main baseline clinical characteristics (eTables 2 and 3 in the Supplement). The majority of patients were white (13 361 [82.8%]), and the median (interquartile range) age was 68 (61-74) years. The majority of tumors were stage I disease (9327 [57.8%]) and adenocarcinoma (11 037 [68.4%]). The median (interquartile range) follow-up time was 24 (10-43) months. There were 2893 patients (17.9%) who had events (deaths from NSCLC) during the follow-up time. There were 1182 patients diagnosed with NSCLC from CHINA database (eFigure 1B in the Supplement). There were 226 events (deaths from NSCLC) over a median (interquartile range) follow-up time of 63.3 (53-70) months. Table 1. Main Characteristics of Patients in the Whole Data Sets of Survival Analysis Data set, No. (%) Characteristic Training SEER (test 1) CHINA (test 2) Age at diagnosis, median (range), y 68 (28-95) 68 (19-92) 60 (30-87) Sex Female 6657 (51.6) 1639 (50.8) 642 (54.3) Male 6255 (48.4) 1589 (49.2) 540 (45.7) Histologic type Adenocarcinoma 8794 (68.1) 2243 (69.5) 948 (80.2) Squamous cell carcinoma 4118 (31.9) 985 (30.5) 234 (19.8) Marital status at diagnosis Unmarried 5304 (41.1) 1843 (57.1) 526 (44.5) Married 7608 (58.9) 1385 (42.9) 656 (55.5) T1a 563 (4.4) 139 (4.3) 128 (10.8) T1b 3156 (24.4) 804 (24.9) 396 (33.5) T1c 2342 (18.1) 641 (19.9) 346 (29.3) T2a 3258 (25.2) 791 (24.5) 208 (17.6) T2b 594 (4.6) 141 (4.4) 56 (4.7) T3 1994 (15.4) 445 (13.8) 40 (3.4) T4 1005 (7.8) 267 (8.3) 8 (0.7) N0 9712 (75.2) 2439 (75.6) 1030 (87.1) N1 1732 (13.4) 418 (12.9) 54 (4.6) N2 1422 (11) 356 (11) 98 (8.3) N3 46 (0.4) 15 (0.5) 0 M0 12 559 (97.3) 3132 (97) 1182 (100) M1a 143 (1.1) 41 (1.3) 0 M1b 202 (1.6) 52 (1.6) 0 M1c 8 (0.1) 3 (0.1) 0 LCCS Alive 10 581 (81.9) 2666 (82.6) 956 (80.9) Dead 2331 (18.1) 562 (17.4) 226 (19.1) Surgery to primary site Abbreviations: LCCS, lung cancer–specific survival; M, Pneumonectomy 613 (4.7) 132 (4.1) 40 (3.4) metastasis; N, node; SEER, Surveillance, Epidemiology, Lobectomy 10 766 (83.4) 2695 (83.5) 872 (73.8) and End Results cancer registry; T, tumor. Sublobar 1444 (11.2) 369 (11.4) 270 (22.8) Other detailed clinical characteristics can be found in None 89 (0.7) 32 (1.0) 0 eTables 2 and 3 in the Supplement. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 5/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Training Curves eFigure 2 in the Supplement demonstrated the training curves of networks in 3 submodels. The accuracy during the training course was indicated by validation and training lines. The curve was plotted to monitor the training course as the weights of the network were adjusted over each epoch, which represented the algorithm runs through the entire training and test data sets. After fine tuning, the change trend of loss and accuracy tended to become smoother and the algorithm maintained high accuracy on the validation set without significant overfitting. With a 500-epoch limit, we chose the model with the best performance on the test data set. Calibration and Validation of the Prognostic DeepSurv for LCSS in the Test Set We compared the TNM staging model to DeepSurv for LCSS in the test data sets (Table 2). The calibration plot indicated the calibration and how far the predictions of DeepSurv deviated from the actual event (Figure 2). In general, the actual outcomes in our databases of all patients with NSCLC for 3-year and 5-year LCSS were highly consistent with those predicted by the DeepSurv model, with most points falling almost directly on the 45° line. The DeepSurv model generated significantly better predictions than the TNM staging model (C statistic for TNM stage vs DeepSurv = 0.70; 95% CI, 0.681-0.731 vs 0.739; 95% CI, 0.713-0.764 [P < .001]). In the test group (CHINA data set), the DeepSurv model (C statistic = 0.742; 95% CI, 0.709-0.775) showed significantly better prediction than TNM model (C statistic = 0.706; 95% CI, 0.681-0.731; P < .001). High C statistic was observed for the results of the lobectomy and sublobar resection test data sets (Table 2). The feature component weightings in DeepSurv model are listed at eTable 4 in the Supplement. The Cox proportional hazard regression model (eTable 5 in the Supplement) was compared with the DeepSurv model for LCSS. The DeepSurv model had significantly better predictions compared with the Cox proportional hazard regression model (C statistic for Cox proportional hazard regression model vs DeepSurv model = 0.716; 95% CI, 0.705-0.727 vs 0.739; 95% CI, 0.713-0.764). The Akaike information criterion value of TNM stage model, Cox proportional hazard regression model, and DeepSurv model were 10741.89, 10307.08, and 10310.52, respectively. Treatment Recommender First, we plotted 2 Kaplan-Meier survival curves: the outcome of test cases whose actual treatments were the same as those recommended and those whose were not (eFigure 3 in the Supplement). The population that followed the recommendation experienced significantly better survival rates than those who did not (SEER: hazard ratio [HR], 2.99; 95% CI, 2.49-3.59; P < .001 vs CHINA: HR, 2.14; 95% CI, 1.65-2.77; P < .001). Furthermore, patients in the test data sets were classified into lobectomy and sublobar resection groups according to the received treatment. Consistent with prior analyses, LCSS favored lobectomy compared with sublobar resection in the subgroup with the lobectomy recommendation (SEER: HR, 1.79; 95% CI, 1.28-2.50; P = .001 vs CHINA: HR, 1.92; 95% CI, 1.30-2.83; P = .001). No significant distinction in survival results were observed for lobectomy and sublobar resection in the subgroup with the sublobar resection recommendation (SEER: HR, 0.65; 95% CI, 0.41-1.02; P = .06 vs CHINA: HR, 0.75; 95% CI, 0.44-1.77; P = .28). Table 2. Comparison of TNM Stage and DeepSurv Model for Survival Prediction in Test Data Sets LCCS outcome Model C statistic (95% CI) P value SEER TNM 0.706 (0.681-0.731) NA DeepSurv 0.739 (0.713-0.764) <.001 CHINA TNM 0.691 (0.659-0.724) NA Abbreviations: LCCS, lung cancer-specific survival; NA, DeepSurv 0.742 (0.709-0.775) <.001 not applicable; SEER, Surveillance, Epidemiology, and Treatment DeepSurv (lobectomy, SEER) 0.725 (0.698-0.751) NA End Results cancer registry; TNM, tumor, node, and DeepSurv (sublobar resection, SEER) 0.698 (0.672-0.725) NA metastasis. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 6/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Model Visualization In the prediction view, the system invokes a prediction model (Figure 3; Video), and the DeepSurv model is used to predict patients’ survival probability. The analysis results are visualized in a graphic view as a survival curve, which indicates the survival probability of the patient input over time. In the recommendation view, a recommendation model is adopted by the system, which can provide different patient survival probabilities for different treatments (lobectomy or sublobar resection) (Figure 3; Video). The survival curves of lobectomy and sublobar resection are also presented in a graphic view to facilitate visual comparison. Discussion The results of our pilot study proved that the deep learning network model (DeepSurv) performed better than conventional linear regression modeling (TNM staging model) in postoperative outcome prediction for patients with newly diagnosed NSCLC. Also, this model may serve as a useful analytical tool for treatment recommendation in patients with NSCLC, given its evidence of the significant prognostic benefits of following the treatment recommendation, which clearly outweigh those associated with not following the recommendation. Previous studies have reported a series of linear models to predict the survival of patients with 24-27 lung cancer. However, few risk factors have been selected to these models, which is significantly associated with the survival or recurrence. For example, Liang et al constructed a nomogram based on 6 factors. On the other hand, our Cox analysis demonstrated the contribution of 16 factors in the Figure 2. Calibration Plots for Lung Cancer–Specific Survival (LCCS) for the DeepSurv Model A B 3-Year survival in SEER data set 5-Year survival in SEER data set 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 Predicted 3-year LCCS Predicted 5-year LCCS C D 3-Year survival in CHINA data set 5-Year survival in CHINA data set 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 Three-year survival (A) and 5-year survival (B) of Surveillance, Epidemiology, and End Results (SEER) 0.2 0.2 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 data set and 3-year survival (C) and 5-year survival (D) Predicted 3-year LCCS Predicted 5-year LCCS of CHINA data set are shown. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 7/12 Actual 3-year LCCS Actual 3-year LCCS Actual 5-year LCCS Actual 5-year LCCS JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival DeepSurv model. Obviously, a more comprehensive analysis could be performed by a nonlinear deep learning model. After reviewing the most relevant advanced research, we found many studies to have already applied deep learning models in their analytical approaches to surgical oncology 13 28-30 research. However, most studies have focused on diagnostic application, such as radiographic 14,31-35 30,36-39 image automated quantification, digital histopathology image interpretation, or 11,40 biomarker analysis. Examples of published research using deep learning models for prognostic prediction in surgical oncology are rare, to our knowledge. In NSCLC research, the deep learning technique has been applied to digital histopathology image interpretation, driver mutation risk 40 33,41 detection, and image characteristics discrimination, but only a few studies have focused on postoperative outcome prediction or surgical recommendation, to our knowledge. As a new analytic tool, the deep learning network model will likely become more widely applied to support clinical decision-making. The performance of deep learning models in improving treatment outcomes is a key question and requires solid validation in the real world. In an analysis of 1194 patients with NSCLC, Hosny et al evaluated the prognostic signatures of quantitative imaging features, which were extracted using deep learning networks. Based on their study of the TNM stages of postoperative patients, the authors’ main finding was that deep learning networks significantly outperformed previous models. In our study, we selected a larger patient cohort with NSCLC of unselected consecutive cases including I to IV stages for model training and testing, which provide more solid results for interpretation. The advantages of the deep learning network model 42,43 for postoperative outcome prediction in surgical research can be summarized as follows. First, DeepSurv shows improved adaptability to variables with a nonlinear association, which includes real- world clinical factors. Unlike other models, deep learning algorithms can integrate the nonlinear risk functions associated with outcomes. Second, DeepSurv possesses flexibility in dealing with complex clinical factors. DeepSurv models cannot only automatically learn feature representations from uninterpreted clinical data but also analyze censored factors. Also, the predictions of the DeepSurv model have been proven to perform better in big data analysis. Owing to its ability to learn factor representation, the advantages of the DeepSurv model in dealing with both large factors and sample 44-46 size may play a big role in biomedical marker analyses. It is a surgeon’s duty to introduce clinical information to patients. To facilitate discussion of different potential surgical options, surgeons and patients need an informative tool that focuses on Figure 3. User-Friendly Interface of DeepSurv Model, Which Facilitates Survival Prediction and Treatment Recommendation Recommendation Predict Survival Patient Information 1.0 RX Summ-Scope Reg LN Sur Resection Female 0.9 Sex: (2003+): 0.8 0.7 Moderately Grade: CS extension (2004+): 0.6 0.5 White 15 Race Recode: 0.4 CS mets at dx (2004+): 0.3 Lung - Pleural/Elastic Layer 0.2 Marital Status Married Invasion (PL) by H and E or PL0 0.1 Elastic Stain: 0.0 Lung - Separate Tumor Nodules - 012 24 36 48 60 IA1 No seprate tumor Stage: lpsilateral Lung Time (months) Lung - Surgery to Primary Site T1a Lobectomy T Stage: (1988-2015): 1.0 Lung - Surgery to Other 0.9 N0 None N Stage: Regional/Distant Sites (1998+): 0.8 0.7 M0 60 M Stage: Age at diagnosis: 0.6 0.5 0.4 Upper lobe, lung Primary Site: 10 CS tumor size (2004+): 0.3 0.2 Regional nodes examined AD 20 Lobectomy Histologic: 0.1 (1998+): Sublobar 0.0 012 24 36 48 60 Regional nodes positive (1998+): Time (months) JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 8/12 Event Percent Event Percent JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival survival benefits. In real cases, the establishment of a user-friendly graphic interface based on a patient communication framework will be key to effectively conveying results and illustrating complex analyses, including prognostic prediction, treatment recommendation to patients and 44,47 family members, and improving the surgeons’ understanding of deep learning models. With its fast application and convenient operation, the user-friendly graphic interface example established in our study (Figure 3; Video) shows potential for use with any type of surgical care. To date, identifying patients who are appropriate for initial surgical management and conveying individualized prognostic analyses of postoperative outcomes has been an elusive goal. Instead, most published models are guided by patient characteristics to generate prognostic factors and are influenced by biases for different treatments. The DeepSurv model and its user-friendly graphic interface has the potential to address this clinical dilemma and better share individual outcomes following different surgical procedures. Limitations Since the innovation of deep learning models, many limitations have been recognized. First, deep learning network models are computationally expensive to train and validate. The process of predictions can be hard to interpret because the deep learning networks function much like black boxes, which make it difficult to determine how the network arrives at its decisions. We also recognize that single-clinical data sources have limited clinical characteristics compared with the automated quantification of radiographic images. In this study, we examined 127 features of 21 characteristics in the model. Some important factors including preoperative elements were neglected, which makes the recommendation system need more improvements and stay at feasibility trial status. Also, external validation is lacking in this study. Further study is needed to validate the advantages of deep learning networks in survival prediction. Conclusions To our knowledge, this study is the first to explore the performance of a deep learning network that integrates Cox proportional hazards (DeepSurv) in NSCLC and to obtain promising results in outcome prediction. In addition, we demonstrated DeepSurv’s potential to provide personalized treatment recommendations based on real clinical data. ARTICLE INFORMATION Accepted for Publication: March 20, 2020. Published: June 3, 2020. doi:10.1001/jamanetworkopen.2020.5842 Open Access: This is an open access article distributed under the terms of the CC-BY License.©2020SheYetal. JAMA Network Open. Corresponding Author: Chang Chen, MD, PhD (changchenc@tongji.edu.cn), and Yijiu Ren, MD (yjscott@hotmail.com), Department of Thoracic Surgery, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai 200443, China. Author Affiliations: Department of Thoracic Surgery, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, China (She, Wu, Deng, Zhang, Su, Jiang, Xie, Ren, Chen); College of Design and Innovation, Tongji University, Shanghai, China (Jin, Cao); Shanghai Key Laboratory of Tuberculosis, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, China (Liu); Computer Science, NYU Shanghai, Shanghai, China (Cao). Author Contributions: Dr Chen had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs She, Wu and Jin equally contributed to this work. Concept and design: Deng, Liu, Ren, Chen. Acquisition, analysis, or interpretation of data: She, Jin, Wu, Deng, Zhang, Su, Jiang, Xie, Cao, Ren, Chen. Drafting of the manuscript: Wu, Zhang, Jiang, Ren, Chen. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 9/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Critical revision of the manuscript for important intellectual content: She, Jin, Deng, Su, Liu, Xie, Cao, Ren, Chen. Statistical analysis: She, Jin, Wu, Deng, Zhang, Xie, Ren, Chen. Obtained funding: She, Xie, Ren, Chen. Administrative, technical, or material support: She, Jin, Su, Jiang, Xie, Ren, Chen. Supervision: Xie, Cao, Ren, Chen. Conflict of Interest Disclosures: Dr Chen reported grants from Shanghai Hospital Development Center during the conduct of the study. No other disclosures were reported. Funding/Support: This work was supported by projects from Shanghai Hospital Development Center (SHDC12012716), Shanghai Municipal Health Commission (2018ZHYL0102), Tongji University AI Program (12712150026), and Shanghai Pulmonary Hospital Innovation Program (FKCX1906). Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. REFERENCES 1. Chen W, Zheng R, Baade PD, et al. Cancer statistics in China, 2015. CA Cancer J Clin. 2016;66(2):115-132. doi:10. 3322/caac.21338 2. Asamura H, Chansky K, Crowley J, et al. The International Association for the Study of Lung Cancer Staging Project: proposals for the revision of the n descriptors in the forthcoming 8th edition of the TNM classification for lung cancer. J Thorac Oncol. 2015;10(12):1675-1684. doi:10.1097/JTO.0000000000000678 3. Chansky K, Sculier JP, Crowley JJ, Giroux D, Van Meerbeeck J, Goldstraw P; International Staging Committee and Participating Institutions. The International Association for the Study of Lung Cancer Staging Project: prognostic factors and pathologic TNM stage in surgically managed non-small cell lung cancer. J Thorac Oncol. 2009;4(7): 792-801. doi:10.1097/JTO.0b013e3181a7716e 4. Kawaguchi T, Takada M, Kubo A, et al. Performance status and smoking status are independent favorable prognostic factors for survival in non-small cell lung cancer: a comprehensive analysis of 26,957 patients with NSCLC. J Thorac Oncol. 2010;5(5):620-630. doi:10.1097/JTO.0b013e3181d2dcd9 5. Sculier JP, Chansky K, Crowley JJ, Van Meerbeeck J, Goldstraw P; International Staging Committee and Participating Institutions. The impact of additional prognostic factors on survival and their relationship with the anatomical extent of disease expressed by the 6th Edition of the TNM Classification of Malignant Tumors and the proposals for the 7th Edition. J Thorac Oncol. 2008;3(5):457-466. doi:10.1097/JTO.0b013e31816de2b8 6. Liang W, Zhang L, Jiang G, et al. Development and validation of a nomogram for predicting survival in patients with resected non-small-cell lung cancer. J Clin Oncol. 2015;33(8):861-869. doi:10.1200/JCO.2014.56.6661 7. Xie D, Marks R, Zhang M, et al. Nomograms predict overall survival for patients with small-cell lung cancer incorporating pretreatment peripheral blood markers. J Thorac Oncol. 2015;10(8):1213-1220. doi:10.1097/JTO. 8. Gold JS, Gönen M, Gutiérrez A, et al. Development and validation of a prognostic nomogram for recurrence-free survival after complete surgical resection of localised primary gastrointestinal stromal tumour: a retrospective analysis. Lancet Oncol. 2009;10(11):1045-1052. doi:10.1016/S1470-2045(09)70242-6 9. Callegaro D, Miceli R, Bonvalot S, et al. Development and external validation of two nomograms to predict overall survival and occurrence of distant metastases in adults after surgical resection of localised soft-tissue sarcomas of the extremities: a retrospective analysis. Lancet Oncol. 2016;17(5):671-680. doi:10.1016/S1470-2045 (16)00010-3 10. Randall RL, Cable MG. Nominal nomograms and marginal margins: what is the law of the line? Lancet Oncol. 2016;17(5):554-556. doi:10.1016/S1470-2045(16)00072-3 11. Li B, Cui Y, Diehn M, Li R. Development and validation of an individualized immune prognostic signature in early-stage nonsquamous non-small cell lung cancer. JAMA Oncol. 2017;3(11):1529-1537. doi:10.1001/jamaoncol. 2017.1609 12. Kopecky KE, Urbach D, Schwarze ML. Risk calculators and decision aids are not enough for shared decision making. JAMA Surg. 2019;154(1):3-4. 13. Hashimoto DA, Rosman G, Rus D, Meireles OR. Artificial intelligence in surgery: promises and perils. Ann Surg. 2018;268(1):70-76. doi:10.1097/SLA.0000000000002693 14. Hosny A, Parmar C, Coroller TP, et al. Deep learning for lung cancer prognostication: a retrospective multi- cohort radiomics study. PLoS Med. 2018;15(11):e1002711. doi:10.1371/journal.pmed.1002711 JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 10/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival 15. Bergquist SL, Brooks GA, Keating NL, Landrum MB, Rose S. Classifying lung cancer severity with ensemble machine learning in health care claims data. Proc Mach Learn Res. 2017;68:25-38. 16. Corey KM, Kashyap S, Lorenzi E, et al. Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): a retrospective, single-site study. PLoS Med. 2018;15(11):e1002701. doi:10.1371/journal.pmed.1002701 17. Matsuo K, Purushotham S, Jiang B, et al. Survival outcome prediction in cervical cancer: Cox models vs deep- learning model. Am J Obstet Gynecol. 2019;220(4):381.e1-381.e14. doi:10.1016/j.ajog.2018.12.030 18. Katzman JL, Shaham U, Cloninger A, Bates J, Jiang T, Kluger Y. DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Med Res Methodol. 2018; 18(1):24. doi:10.1186/s12874-018-0482-1 19. Shi S, Hua J, Liang C, et al. Proposed modification of the 8th edition of the AJCC staging system for pancreatic ductal adenocarcinoma. Ann Surg. 2019;269(5):944-950. doi:10.1097/SLA.0000000000002668 20. Wong SM, Freedman RA, Sagara Y, Aydogan F, Barry WT, Golshan M. Growing use of contralateral prophylactic mastectomy despite no improvement in long-term survival for invasive breast cancer. Ann Surg. 2017;265(3): 581-589. doi:10.1097/SLA.0000000000001698 21. Dai C, Ren Y, Xie D, et al. Does lymph node metastasis have a negative prognostic impact in patients with NSCLC and M1a disease? J Thorac Oncol. 2016;11(10):1745-1754. doi:10.1016/j.jtho.2016.06.030 22. Leung K-M, Elashoff RM, Afifi AA. Censoring issues in survival analysis. Annu Rev Public Health. 1997;18(1): 83-104. doi:10.1146/annurev.publhealth.18.1.83 23. GitHub. thoraciclang/Deep_Lung. Accessed May 11, 2020. https://github.com/thoraciclang/Deep_Lung 24. Wang Y, Qu X, Kam N-W, et al. An inflammation-related nomogram for predicting the survival of patients with non-small cell lung cancer after pulmonary lobectomy. BMC Cancer. 2018;18(1):692-692. doi:10.1186/s12885- 018-4513-4 25. Liang W, Zhang L, Jiang G, et al. Development and validation of a nomogram for predicting survival in patients with resected non-small-cell lung cancer. J Clin Oncol. 2015;33(8):861-869. doi:10.1200/JCO.2014.56.6661 26. Yap W-K, Shih M-C, Kuo C, et al. Development and validation of a nomogram for assessing survival in patients with metastatic lung cancer referred for radiotherapy for bone metastases. JAMA Netw Open. 2018;1(6): e183242-e183242. doi:10.1001/jamanetworkopen.2018.3242 27. Zhang Y, Zheng D, Xie J, et al. Development and validation of web-based nomograms to precisely predict conditional risk of site-specific recurrence for patients with completely resected non-small cell lung cancer: a multiinstitutional study. Chest. 2018;154(3):501-511. doi:10.1016/j.chest.2018.04.040 28. Carin L, Pencina MJ. On deep learning for medical image analysis. JAMA. 2018;320(11):1192-1193. doi:10.1001/ jama.2018.13316 29. Liang H, Tsui BY, Ni H, et al. Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. Nat Med. 2019;25(3):433-438. doi:10.1038/s41591-018-0335-9 30. Kather JN, Pearson AT, Halama N, et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med. 2019;25(7):1054-1056. doi:10.1038/s41591-019-0462-y 31. De Fauw J, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018;24(9):1342-1350. doi:10.1038/s41591-018-0107-6 32. Peake MD, Navani N, Baldwin DR. The continuum of screening and early detection, awareness and faster diagnosis of lung cancer. Thorax. 2018;73(12):1097-1098. doi:10.1136/thoraxjnl-2018-212189 33. Nam JG, Park S, Hwang EJ, et al. Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology. 2019;290(1):218-228. doi:10.1148/ radiol.2018180237 34. Hwang EJ, Park S, Jin KN, et al; DLAD Development and Evaluation Group. Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open. 2019;2(3):e191095. doi:10.1001/jamanetworkopen.2019.1095 35. Topalovic M, Das N, Burgel PR, et al; Pulmonary Function Study Investigators. Pulmonary Function Study Investigators. Artificial intelligence outperforms pulmonologists in the interpretation of pulmonary function tests. Eur Respir J. 2019;53(4):1801660. doi:10.1183/13993003.01660-2018 36. Ren J, Karagoz K, Gatza ML, et al. Recurrence analysis on prostate cancer patients with Gleason score 7 using integrated histopathology whole-slide images and genomic data through deep neural networks. J Med Imaging (Bellingham). 2018;5(4):047501. doi:10.1117/1.JMI.5.4.047501 JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 11/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival 37. Campanella G, Hanna MG, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med. 2019;25(8):1301-1309. doi:10.1038/s41591-019-0508-1 38. Gertych A, Swiderska-Chadaj Z, Ma Z, et al. Convolutional neural networks can accurately distinguish four histologic growth patterns of lung adenocarcinoma in digital slides. Sci Rep. 2019;9(1):1483. doi:10.1038/s41598- 018-37638-9 39. Steiner DF, MacDonald R, Liu Y, et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol. 2018;42(12):1636-1646. doi:10.1097/PAS. 40. Riesselman AJ, Ingraham JB, Marks DS. Deep generative models of genetic variation capture the effects of mutations. Nat Methods. 2018;15(10):816-822. doi:10.1038/s41592-018-0138-4 41. Tanaka N, Kanatani S, Tomer R, et al. Whole-tissue biopsy phenotyping of three-dimensional tumours reveals patterns of cancer heterogeneity. Nat Biomed Eng. 2017;1(10):796-806. doi:10.1038/s41551-017-0139-0 42. Yasaka K, Abe O. Deep learning and artificial intelligence in radiology: current applications and future directions. PLoS Med. 2018;15(11):e1002707. doi:10.1371/journal.pmed.1002707 43. Smith CC, Chai S, Washington AR, et al. Machine-learning prediction of tumor antigen immunogenicity in the selection of therapeutic epitopes. Cancer Immunol Res. 2019;7(10):1591-1604. doi:10.1158/2326-6066.CIR- 19-0155 44. Wang F, Casalino LP, Khullar D. Deep learning in medicine-promise, progress, and challenges. JAMA Intern Med. 2019;179(3):293-294. 45. Shaffie A, Soliman A, Fraiwan L, et al. A generalized deep learning-based diagnostic system for early diagnosis of various types of pulmonary nodules. Technol Cancer Res Treat. 2018;17:1533033818798800. doi:10.1177/ 46. Kantz ED, Tiwari S, Watrous JD, Cheng S, Jain M. Deep neural networks for classification of LC-MS spectral peaks. Anal Chem. 2019;91(19):12407-12413. doi:10.1021/acs.analchem.9b02983 47. Simon G, DiNardo CD, Takahashi K, et al. Applying Artificial Intelligence to Address the Knowledge Gaps in Cancer Care. Oncologist. 2019;24(6):772-782. 48. Coiera E. On algorithms, machines, and medicine. Lancet Oncol. 2019;20(2):166-167. doi:10.1016/S1470-2045 (18)30835-0 SUPPLEMENT. eMethods. eTable 1. All Clinical Features Integrated in the Model eTable 2. Characteristics of Patients in the Training Dataset of Survival Analysis eTable 3. Characteristics of Patients in the Test Dataset of Survival Analysis eTable 4. Feature Component Weightings in the DeepSurv Model eTable 5. Survival Predictors in Cox PH Model eFigure 1. Flow chart of datasets construction. (A) SEER dataset, (B) CHINA dataset eFigure 2. Training curves of networks in the survival dataset of SEER database (A), lobectomy dataset (B), and sublobar resection dataset (C) eFigure 3. Lung Cancer–Specific Survival Recommendation Comparisons of SEER Data set (A), SEER Lobectomy Test Data set (B), and SEER Sublobar Resection Test Data set (C); Lung Cancer–Specific Survival Recommendation Comparisons of CHINA Data set (D), CHINA Lobectomy Test Data set (E), and CHINA sublobar Resection Test Data set (F) JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 12/12 eMethods. eTable 1. All Clinical Features Integrated in the Model Characteristics Features Sex Female Male Main Lower Overlapping Lung, bronchus/Carin Upper lobe, Primary Site Middle lobe, lung lobe, lesion of NOS/Bronch a/Hilum/Bronch lung lung lung us, NOS us intermedius Histologic Type AD SD Grade Moderately Poor undifine well Stage IA1 IA2 IA3 IB IIA IIB IIIA IIIB IIIC IVA IVB T stage T1a T1b T1c T2a T2b T3 T4 N stage N0 N1 N2 N3 M stage M0 M1a M1b M1c RX Summ--Scope System Reg LN Sur None Biopsy Resection resection (2003+) Microscopic focus or foci CS tumor size Diffuse (entire 1-988mm only and no (2004+) lobe) size of focus given Blood Tumor vessel(s) of/involvi Direct , major + ng main extensio Direct stem n extensio bronchus Extensio Tumor to:Brachi Superior n less than n proven al sulcus to:Brachi 2.0 cm to:Pleura by plexus, tumor al from , visceral presence inferior WITH plexus, carina + or NOS of Atelectasis/obs Extensio branches encasem inferior Atelectas (WITHO Tumor malignan tructive n Stated or NOS, ent of branches Superficial tumor Extensio Invasion Tumor is/obstru UT of/involvi t cells in Stated pneumonitis to:Pleura Stated Stated as T2 Atelectas Stated from subclavi Multiple or NOS, Pleural Stated Stated of any size with n from Direct Extension to but of of/involvi ctive pleural ng main sputum as T1a Stated as Tumor that extends to , visceral as T2a as T2b [NOS] is/obstru as T3 superior an masses/ tumor as T4 as T4 from invasive Stated as other tumor not into pleura, pleura, ng main pneumon effusion) stem Parietal Aorta + Adjacent Malignan Further or with no T1b with no involving the hilar region Invasion or NOS with no with no with no ctive with no Invasion sulcus/C vessels separate superior Heart/Vis foci Adjacent with no with no In situ, Tumor component limited T1[NOS] with parts of invasion Tumor Tumor including invasion including Pulmona stem itis that Pulmona bronchus pericardi Tumor Blood Malignan Heart/Vis Aorta + Vertebra Inferior rib + t contiguo bronchial CS extension other other main Localize but does not of (WITHO other other other pneumon other of hest OR tumor sulcus/C ceral Adjacent separate rib other other to bronchial wall, lung to invasion bronchus Aorta intraepithelial, confined to no other into an confined confined of elastic ry extends ry less than um or confined vessel(s) t pleural ceral Adjacent vena Blood pericardi us washing (2004+) informati information stem d, NOS involve the pleura, UT informati informati informati itis informati phrenic (thoracic WITH nodule(s hest pericardi rib from +other informati informati noninvasive one lung with or without information main adjacent to hilus to carina layer/BUT not through ligament less than to the ligament 2.0 cm pericardi to carina , major effusion pericardi rib foramina cava vessel(s) al extensio s but not on on on bronchus entire lung Or NOS pleural on on on on on on involving on on nerve ) unequivo ) in the (thoracic um direct extention on on on on proximal extension on extension stem ipsilatera through the elastic the 2.0 cm hilar + Tumor from um, NOS um , major effusion n visualize extensio extension , NOS atelectasis/obs effusion) extensio extensio size or entire extensio wall/Diap cal SAME pleural extensio extensio to the main stem bronchus l lobe layer elastic from region of/involvi carina + d by n tructive Pulmona n n extensio lung n hragm/P involvem lobe wall/Diap invasion n n bronchus , NOS layer carina but does ng main Invasion imaging pneumonitis, ry n ancoast ent of hragm/P not stem of pleura or NOS ligament tumor superior ancoast involve bronchus bronchos (superior branches tumor the less than copy; sulcus of (superior entire 2.0 cm "occult" syndrom brachial sulcus lung Or from carcinom e), plexus syndrom atelectas carina a NOS/Par e), is/obstru ietal NOS/Par ctive pleura ietal pneumon pleura/S itis, NOS uperior Extensio to:Skelet Abdomin Extensio al al n muscle/S organs+ to:Contra ternum/S Distant Contralat lateral kin of metastas eral lung/Con Extensio chest Extensio es + lung/Con tralateral n to +Contral n Distant tralateral Distant Malignan main Pleural contralat ateral to:Contra node(s) Malignant main metastas Distant Stated t stem tumor eral lung Stated lung/Con lateral + Stated pleural Separate Distant Extensio stem Distant is plus metastas as M1 Malignan Malignant pericardi bronchus foci or plus as M1a Distant tralateral lung/Con Contralat as M1b effusion, tumor lymph n bronchus metastas Distant Distant distant es plus [NOS] Distant lymph t pleural pleural Malignan al /Separat nodules pleural with no lymph main tralateral eral with no ipsilateral Distant lymph Distant lymph nodule(s nodes to:Skelet /Separat is plus metastas metastas lymph distant with no node(s), Malignant pleural effusion, effusion, t effusion e tumor on the or other nodes stem main lung/Con other CS mets at dx No distant and node(s), nodes plus pleural ) in plus al Abdomin e tumor pleural is plus is plus nodes lymph other including effusion, ipsilateral contralat unknown if pericardi plus nodule(s ipsilatera pericardi informati plus bronchus stem tralateral informati (2004+) metastasis contralateral including or pericardial different extensio muscle/S al organs nodule(s or pleural distant plus nodes informati cervical or same lung eral or ipsilateral or al contralat ) in l lung al on on pleural /Separat bronchus main on on lungs cervical nodes effusion lobe, n to ternum/S ) in pericardi tumor lymph pleural plus on on nodes other contralateral effusion eral or contralat separate effusion distant tumor e tumor /Separat stem distant (Bilateral same contralat kin of contralat al foci node(s) or pleural distant lung lung bilateral eral from or metastas foci nodule(s e tumor bronchus metastas pleural lung eral lung chest eral effusion pericardi tumor metastas pleural lung/Ple direct separate is ) in nodule(s /Separat is effusion) lung/Ple al foci is effusion ural invasion pleural contralat ) in e tumor ural effusion tumor tumor eral contralat nodule(s tumor foci or foci lung/Ple eral lung ) in foci or nodules ural contralat nodules on tumor eral lung on contralat foci or contralat eral lung nodules eral lung on contralat eral lung Regional nodes 1-100 examined (1988+) Regional nodes 1-100 positive (1988+) American Asian or African Race recode White Indian/Alaska Pacific Americans Native Islander Age at diagnosis 1-120 Marital status at diagnosis married un-married married=1 un- married=0 Lung - Pleural/Elastic Layer Invasion PL0 PL1 PL2 PL3 (PL) by H and E or Elastic Stain Separate tumor Separate nodules, Separate tumor Lung - Separate No separate tumor nodules ipsilatera nodules in Tumor Nodules - tumor nodules in ipsilateral l lung, ipsilateral lung, Ipsilateral Lung noted lung, different same same lobe lobe and different lobe Lung - Surgery to Primary Site Peumonectomy Lobectomy Sublobar None (1988-2015) Lung - Surgery to Surgery To Other Surgery To None Distant Site Regional/Distant Regional Site Or Nodes Sites (1998+) eTable 2. Characteristics of Patients in the Training Dataset of Survival Analysis IA1 IA2 IA3 IB IIA IIB IIIA IIIB IIIC IVA IVB total Stage median range median range median range median range mediarange mediarange median range median range median range medianrange median range median range Age at diagnosis 67 36-90 67 29-90 69 31-92 69 32-92 69 35-86 67 30-95 67 29-90 66 34-89 64 46-85 65 28-84 61 50-78 68 28-95 count % count % count % count % count % count % count % count % count % count % count % count % Sex Female 324 62.5 1585 57.5 1079 56.7 1158 51.3 180 45 1025 44.9 908 47 213 42.9 5 29.4 175 50.7 5 62.5 6657 51.6 Male 194 37.5 1172 42.5 823 43.3 1098 48.7 220 55 1258 55.1 1022 53 283 57.1 12 70.6 170 49.3 3 37.5 6255 48.4 Histologic Type 2 AD 410 79.2 2110 76.5 1347 70.8 1492 66.1 223 55.8 1384 60.6 1245 64.5 307 61.9 10 58.8 260 75.4 6 75 8794 68.1 SD 108 20.8 647 23.5 555 29.2 764 33.9 177 44.3 899 39.4 685 35.5 189 38.1 7 41.2 85 24.6 2 25 4118 31.9 Marital status at diagnosis un-married 203 39.2 1149 41.7 805 42.3 970 43 163 40.8 931 40.8 753 39 191 38.5 4 23.5 133 38.6 2 25 5304 41.1 married 315 60.8 1608 58.3 1097 57.7 1286 57 237 59.3 1352 59.2 1177 61 305 61.5 13 76.5 212 61.4 6 75 7608 58.9 T1a 518 100 0 0 0 0 0 0 0 0 22 1 21 1.1 0 0 0 0 2 0.6 0 0 563 4.4 T1b 0 0 2757 100 0 0 0 0 0 0 219 9.6 146 7.6 5 1 0 0 29 8.4 0 0 3156 24.4 T1c 0 0 0 0 1902 100 0 0 0 0 217 9.5 182 9.4 4 0.8 0 0 36 10.4 1 12.5 2342 18.1 T2a 0 0 0 0 0 0 2256 100 0 0 499 21.9 425 22 4 0.8 0 0 73 21.2 1 12.5 3258 25.2 T2b 0 0 0 0 0 0 0 0 400 100 111 4.9 71 3.7 1 0.2 0 0 11 3.2 0 0 594 4.6 T3 0 0 0 0 0 0 0 0 0 0 1214 53.2 381 19.7 284 57.3 11 64.7 102 29.6 2 25 1994 15.4 T4 0 0 0 0 0 0 0 0 0 0 1 0 704 36.5 198 39.9 6 35.3 92 26.7 4 50 1005 7.8 N0 518 100 2757 100 1902 100 2256 100 400 100 1216 53.3 484 25.1 0 0 0 0 177 51.3 2 25 9712 75.2 N1 0 0 0 0 0 0 0 0 0 0 1067 46.7 601 31.1 0 0 0 0 63 18.3 1 12.5 1732 13.4 N2 0 0 0 0 0 0 0 0 0 0 0 0 845 43.8 482 97.2 0 0 92 26.7 3 37.5 1422 11 N3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14 2.8 17 100 13 3.8 2 25 46 0.4 M0 518 100 2757 100 1902 100 2256 100 400 100 2283 100 1930 100 496 100 17 100 0 0 0 0 12559 97.3 M1a 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 143 41.4 0 0 143 1.1 M1b 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 202 58.6 0 0 202 1.6 M1c 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 100 8 0.1 LCCS Alive 496 95.8 2577 93.5 1701 89.4 1950 86.4 340 85 1758 77 1303 67.5 279 56.3 7 41.2 169 49 1 12.5 10581 81.9 dead 22 4.2 180 6.5 201 10.6 306 13.6 60 15 525 23 627 32.5 217 43.8 10 58.8 176 51 7 87.5 2331 18.1 Pleural/Elastic Layer Invasion PL0 518 100 2757 100 1902 100 1369 60.7 399 99.8 1715 75.1 1450 75.1 327 65.9 14 82.4 239 69.3 8 100 10698 82.9 PL1 0 0 0 0 0 0 489 21.7 1 0.3 221 9.7 181 9.4 67 13.5 1 5.9 37 10.7 0 0 997 7.7 PL2 0 0 0 0 0 0 398 17.6 0 0 157 6.9 194 10.1 57 11.5 1 5.9 43 12.5 0 0 850 6.6 PL3 0 0 0 0 0 0 0 0 0 0 190 8.3 105 5.4 45 9.1 1 5.9 26 7.5 0 0 367 2.8 Separate Tumor Nodules 1 518 100 2757 100 1902 100 2255 100 400 100 1903 83.4 1617 83.8 348 70.2 14 82.4 241 69.9 3 37.5 11958 92.6 2 0 0 0 0 0 0 0 0 0 0 0 0 158 8.2 47 9.5 0 0 42 12.2 3 37.5 250 1.9 3 0 0 0 0 0 0 1 0 0 0 380 16.6 126 6.5 91 18.3 2 11.8 45 13 1 12.5 646 5 4 0 0 0 0 0 0 0 0 0 0 0 0 29 1.5 10 2 1 5.9 17 4.9 1 12.5 58 0.4 Surgery to Other Regional/Distant Sites None 515 99.4 2748 99.7 1892 99.5 2236 99.1 399 99.8 2252 98.6 1885 97.7 480 96.8 14 94.1 275 79.7 7 87.5 12705 98.4 Distant 0 0 4 0.1 4 0.2 7 0.3 0 0 11 0.5 11 0.6 4 0.8 0 0 64 18.6 1 12.5 106 0.8 Regional 3 0.6 5 0.2 6 0.3 13 0.6 1 0.3 20 0.9 34 1.8 12 2.4 1 5.9 6 1.7 0 0 101 0.8 eTable 3. Characteristics of Patients in the Test Dataset of Survival Analysis Stage IA1 IA2 IA3 IB IIA IIB IIIA IIIB IIIC IVA IVB total median range median range median range median range median range median range median range median range median range median range median range median range Age at diagnosis 68 34-86 67 35-90 69 36-92 69 29-90 68 40-87 68 42-91 67 39-88 66 19-87 63 36-68 65 40-83 67 59-69 68 19-92 count % count % count % count % count % count % count % count % count % count % count % count % Sex 3 3228 Female 82 66.1 407 57.2 267 52 278 50.9 35 35.4 267 50.1 203 41.7 51 44 2 66.7 45 48.4 2 66.7 1639 50.8 Male 42 33.9 304 42.8 246 48 268 49.1 64 64.6 266 49.9 284 58.3 65 56 1 33 48 51.6 1 33.3 1589 49.2 Histologic Type 2 AD 99 79.8 541 76.1 369 71.9 380 69.6 48 48.5 346 64.9 319 65.5 73 62.9 2 66.7 64 68.8 2 66.7 2243 69.5 SD 25 20.2 107 23.9 144 28.1 166 30.4 51 51.5 187 35.1 168 34.5 43 37.1 1 33.3 29 31.2 1 33.3 985 30.5 Marital status at diagnosis married 59 47.6 308 43.3 294 57.3 305 55.9 40 40.4 295 55.3 295 60.6 69 59.5 1 33.3 56 60.2 1 33.3 1843 57.1 un-married 65 52.4 403 56.7 219 42.7 241 44.1 59 59.6 238 44.7 192 39.4 47 40.5 2 66.7 37 39.8 2 66.7 1385 42.9 T1a 124 100 0 0 0 0 0 0 0 0 7 1.3 7 1.4 0 0 0 0 4 1.1 0 0 139 4.3 T1b 0 0 711 100 0 0 0 0 0 0 39 7.3 43 8.8 2 1.7 0 0 9 9.7 0 0 804 24.9 T1c 0 0 0 0 513 100 0 0 0 0 65 12.2 52 10.7 2 1.7 0 0 9 9.7 0 0 641 19.9 T2a 0 0 0 0 0 0 546 100 0 0 126 23.6 97 19.9 2 1.7 0 0 20 21.5 0 0 791 24.5 T2b 0 0 0 0 0 0 0 0 99 100 18 3.4 19 3.9 0 0 0 0 5 5.4 0 0 141 4.4 T3 0 0 0 0 0 0 0 0 0 0 278 52.2 76 15.6 67 57.8 1 33.3 22 23.7 1 33.3 445 13.8 T4 0 0 0 0 0 0 0 0 0 0 0 0 193 39.6 43 37.1 2 66.7 27 29 2 66.7 267 8.3 N0 124 100 711 100 513 100 546 100 99 100 278 52.2 120 24.6 0 0 0 0 47 50.5 1 33.3 2439 75.6 N1 0 0 0 0 0 0 0 0 0 0 255 47.8 149 30.6 0 0 0 0 13 14 1 33.3 418 12.9 N2 0 0 0 0 0 0 0 0 0 0 0 0 218 44.8 110 94.8 0 0 27 29 1 33.3 356 11 N3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 5.2 4 100 6 6.5 0 0 15 0.5 M0 124 100 711 100 513 100 546 100 99 100 533 100 487 100 116 100 3 100 0 0 0 0 3132 97 M1a 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 41 44.1 0 0 41 1.3 M1b 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 52 55.9 0 0 52 1.6 M1c 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 100 3 0.1 LCCS Alive 121 97.6 657 92.4 470 91.6 478 87.5 91 91.9 413 77.5 321 65.9 66 56.9 1 33.3 47 50.5 1 33.3 2666 82.6 dead 3 2.4 54 7.6 43 8.4 68 12.5 8 8.1 120 22.5 166 34.1 50 43.1 2 66.7 46 49.5 2 66.7 562 17.4 Pleural/Elastic Layer Invasion PL0 124 100 711 100 513 100 315 57.7 99 100 405 76 360 73.9 80 69 2 66.7 66 71 1 33.3 2676 82.9 PL1 0 0 0 0 0 0 133 24.4 0 0 47 8.8 50 10.3 10 8.6 0 0 8 8.6 1 33.3 249 7.7 PL2 0 0 0 0 0 0 98 17.9 0 0 37 6.9 52 10.7 17 14.7 1 33.3 15 16.1 1 33.3 221 6.8 PL3 0 0 0 0 0 0 0 0 0 0 44 8.3 25 5.1 9 7.8 0 0 4 4.3 0 0 82 2.5 Separate Tumor Nodules 1 124 100 711 100 513 100 546 100 99 100 433 83.1 410 84.2 82 70.7 3 100 69 74.2 2 66.7 3002 93 2 0 0 0 0 0 0 0 0 0 0 0 0 36 7.4 10 8.6 0 0 10 10.8 1 33.3 57 1.8 3 0 0 0 0 0 0 0 0 0 0 90 16.9 28 5.7 23 19.8 0 0 10 10.8 0 0 151 4.7 4 0 0 0 0 0 0 0 0 0 0 0 0 13 2.7 1 0.9 0 0 4 4.3 0 0 18 0.6 Surgery to Other Regional/Distant Sites None 124 100 711 100 513 100 545 99.8 99 100 527 98.9 478 98.2 115 99.1 3 100 74 79.6 2 66.7 3181 98.5 Distant 0 0 0 0 0 0 0 0 0 0 0 0 2 0.4 0 0 0 0 17 18.3 1 33.3 24 0.7 Regional 0 0 0 0 0 0 1 0.2 0 0 6 1.1 7 1.4 1 0.9 0 0 2 2.2 0 0 23 0.7 eTable 4. Feature Component Weightings in the DeepSurv Model Features Weight Features Weight Features Weight Age at diagnosis 0.5722479 CS extension (2004+)=540 0.42365953 Stage=3 -0.0717 CS tumor size (2004+) 0.6724694 CS extension (2004+)=550 0.08433475 Stage=4 -0.07168 Regional nodes examined (1988+) -0.4995487 CS extension (2004+)=560 0.13039102 Stage=5 -0.01987 Regional nodes positive (1988+) 0.7574372 CS extension (2004+)=570 0.06113536 Stage=6 0.041379 Sex=1 -0.062900014 CS extension (2004+)=590 0.22531554 Stage=7 0.101898 Sex=2 0.1671767 CS extension (2004+)=600 0.12043501 Stage=8 0.093093 Histologic Type ICD-O-3=0 0.055728845 CS extension (2004+)=610 -0.90538454 Stage=9 0.104782 Histologic Type ICD-O-3=1 -0.048216447 CS extension (2004+)=680 -0.824522 Stage=10 0.034815 Histologic Type ICD-O-3=2 0.015860233 CS extension (2004+)=700 0.16996412 Stage=11 0.51159 Histologic Type ICD-O-3=3 -0.08446639 CS extension (2004+)=705 -0.9847599 T8=1 0.209211 Histologic Type ICD-O-3=4 0.0417398 CS extension (2004+)=710 0.3103478 T8=2 0.038959 Histologic Type ICD-O-3=5 -0.00887268 CS extension (2004+)=730 -0.20042646 T8=3 0.056467 Histologic Type ICD-O-3=6 0.025923487 CS extension (2004+)=740 0.81883913 T8=4 0.049242 Histologic Type 2=1 -0.028282069 CS extension (2004+)=745 0.40200815 T8=5 0.06238 Histologic Type 2=2 0.020090567 CS extension (2004+)=750 0.16024342 T8=6 -0.09599 Grade=1 -0.002670259 CS extension (2004+)=770 -0.86313397 T8=7 -0.12861 Grade=2 0.063437365 CS extension (2004+)=785 -0.17053518 N8=1 -0.06939 Grade=3 0.15810749 CS mets at dx (2004+)=0 -0.07108595 N8=2 0.052847 Grade=4 -0.124901354 CS mets at dx (2004+)=15 0.14313275 N8=3 0.070122 RX Summ--Scope Reg LN Sur (2003+) -0.05786139 CS mets at dx (2004+)=16 0.08157881 N8=4 0.202186 RX Summ--Scope Reg LN Sur (2003+) 0.029179208 CS mets at dx (2004+)=17 -0.34559816 M8=1 -0.21266 RX Summ--Scope Reg LN Sur (2003+) 0.020770853 CS mets at dx (2004+)=18 0.059344094 M8=2 -0.02519 RX Summ--Scope Reg LN Sur (2003+) -0.03520994 CS mets at dx (2004+)=20 -0.12556794 M8=3 0.030704 CS extension (2004+)=100 -0.0725175 CS mets at dx (2004+)=21 0.99101025 M8=4 0.576268 CS extension (2004+)=110 0.04731824 CS mets at dx (2004+)=23 -0.19575515 Marital status at diagnosis=0 0.034368 CS extension (2004+)=115 0.026038347 CS mets at dx (2004+)=24 -0.037997384 Marital status at diagnosis=1 -0.10559 CS extension (2004+)=120 0.004864246 CS mets at dx (2004+)=25 -0.76219594 Lung - Pleural/Elastic Layer Invasion (PL) by H and E or Elastic Stain=0 0.001433 CS extension (2004+)=125 -0.09474413 CS mets at dx (2004+)=26 0.90065193 Lung - Pleural/Elastic Layer Invasion (PL) by H and E or Elastic Stain=1 0.034629 CS extension (2004+)=200 0.021685144 CS mets at dx (2004+)=30 -0.18796362 Lung - Pleural/Elastic Layer Invasion (PL) by H and E or Elastic Stain=2 0.001899 CS extension (2004+)=210 -0.16040157 CS mets at dx (2004+)=32 0.6542427 Lung - Pleural/Elastic Layer Invasion (PL) by H and E or Elastic Stain=3 0.164463 CS extension (2004+)=220 -0.10082796 CS mets at dx (2004+)=33 -1.1801782 Lung - Separate Tumor Nodules - Ipsilateral Lung=1 -0.03875 CS extension (2004+)=230 -0.21979994 CS mets at dx (2004+)=36 0.88033414 Lung - Separate Tumor Nodules - Ipsilateral Lung=2 0.063924 CS extension (2004+)=300 -0.033153117 CS mets at dx (2004+)=37 -0.1698591 Lung - Separate Tumor Nodules - Ipsilateral Lung=3 0.042076 CS extension (2004+)=400 -0.029358057 CS mets at dx (2004+)=40 0.14898834 Lung - Separate Tumor Nodules - Ipsilateral Lung=4 0.150139 CS extension (2004+)=410 -0.102100626 CS mets at dx (2004+)=41 0.5691616 Lung - Surgery to Primary Site (1988-2015)=1 0.005233 CS extension (2004+)=420 -0.07669448 CS mets at dx (2004+)=42 0.032155376 Lung - Surgery to Primary Site (1988-2015)=2 -0.13327 CS extension (2004+)=430 0.057814617 CS mets at dx (2004+)=43 0.030705813 Lung - Surgery to Primary Site (1988-2015)=3 -0.03041 CS extension (2004+)=440 0.22026922 CS mets at dx (2004+)=51 0.6830787 Lung - Surgery to Primary Site (1988-2015)=4 0.141988 CS extension (2004+)=455 -0.01990803 CS mets at dx (2004+)=52 1 Lung - Surgery to Other Regional/Distant Sites (1998+)=1 -0.04313 CS extension (2004+)=460 -0.061044298 CS mets at dx (2004+)=53 0.6274554 Lung - Surgery to Other Regional/Distant Sites (1998+)=2 0.117256 CS extension (2004+)=465 0.56912345 CS mets at dx (2004+)=70 -0.58639836 Lung - Surgery to Other Regional/Distant Sites (1998+)=3 0.06074 CS extension (2004+)=500 -0.10761972 Stage=1 -0.4348911 CS extension (2004+)=520 0.20671241 Stage=2 -0.15681928 eTable 5. Survival Predictors in Cox PH Model eFigure 1. Flow chart of datasets construction. (A) SEER dataset, (B) CHINA dataset eFigure 2. Training curves of networks in the survival dataset of SEER database (A), lobectomy dataset (B), and sublobar resection dataset (C). The red and purple curves indicate loss of the training and test datasets, respectively; the blue and yellow curves indicate the accuracy of the training and test datasets, respectively. eFigure 3. Lung cancer specific survival recommendation comparisons of SEER dataset (A), SEER lobectomy test dataset (B), and SEER sublobar resection test dataset (C); Lung cancer specific survival recommendation comparisons of CHINA dataset (D), CHINA lobectomy test dataset (E), and CHINA sublobar resection test dataset (F). http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JAMA Network Open American Medical Association

Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival

Loading next page...
 
/lp/american-medical-association/development-and-validation-of-a-deep-learning-model-for-non-small-cell-wQmam1aBCs
Publisher
American Medical Association
Copyright
Copyright 2020 She Y et al. JAMA Network Open.
eISSN
2574-3805
DOI
10.1001/jamanetworkopen.2020.5842
Publisher site
See Article on Publisher Site

Abstract

Key Points Question Can deep learning IMPORTANCE There is a lack of studies exploring the performance of a deep learning survival neural architecture be applied for individual network in non–small cell lung cancer (NSCLC). prognosis evaluation and treatment recommendation? OBJECTIVES To compare the performances of DeepSurv, a deep learning survival neural network Findings In this cohort study of 17 322 with a tumor, node, and metastasis staging system in the prediction of survival and test the reliability patients with non–small cell lung cancer. of individual treatment recommendations provided by the deep learning survival neural network. The performance of a deep learning model was assessed on real-life clinical DESIGN, SETTING, AND PARTICIPANTS In this population-based cohort study, a deep learning– data sets. The ability of a deep learning based algorithm was developed and validated using consecutive cases of newly diagnosed stages I to model to learn complex associations IV NSCLC between January 2010 and December 2015 in a Surveillance, Epidemiology, and End between an individual’s characteristics Results database. A total of 127 features, including patient characteristics, tumor stage, and and the outcome benefits of different treatment strategies, were assessed for analysis. The algorithm was externally validated on an treatments was also elucidated; independent test cohort, comprising 1182 patients with stage I to III NSCLC diagnosed between particularly, a deep learning network January 2009 and December 2013 in Shanghai Pulmonary Hospital. Analysis began January 2018 identified persons with non–small cell and ended June 2019. lung cancer and survival more accurately than tumor, node, metastasis staging. MAIN OUTCOMES AND MEASURES The deep learning survival neural network model was compared with the tumor, node, and metastasis staging system for lung cancer–specific survival. The Meaning Findings suggest that this C statistic was used to assess the performance of models. A user-friendly interface was provided to novel analytical approach may have facilitate the survival predictions and treatment recommendations of the deep learning survival great potential in providing individual neural network model. prognostic information and treatment recommendations in real clinical RESULTS Of 17 322 patients with NSCLC included in the study, 13 361 (77.1%) were white and the scenarios. median (interquartile range) age was 68 (61-74) years. The majority of tumors were stage I disease (10 273 [59.3%]) and adenocarcinoma (11 985 [69.2%]). The median (interquartile range) follow-up Invited Commentary time was 24 (10-43) months. There were 3119 patients who had lung cancer–related death during the follow-up period. The deep learning survival neural network model showed more promising results Video in the prediction of lung cancer–specific survival than the tumor, node, and metastasis stage on the Supplemental content test data set (C statistic = 0.739 vs 0.706). The population who received the recommended treatments had superior survival rates than those who received treatments not recommended Author affiliations and article information are listed at the end of this article. (hazard ratio, 2.99; 95% CI, 2.49-3.59; P < .001), which was verified by propensity score–matched groups. The deep learning survival neural network model visualization was realized by a user-friendly graphic interface. CONCLUSIONS AND RELEVANCE The deep learning survival neural network model shows potential benefits in prognostic evaluation and treatment recommendation with respect to lung (continued) Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 1/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Abstract (continued) cancer–specific survival. This novel analytical approach may provide reliable individual survival information and treatment recommendations. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 Introduction Lung cancer is the most commonly diagnosed cancer in China and the second in the United States, approximately 85% of which is non–small cell lung cancer (NSCLC). The precise stratification of patients with NSCLC into groups according to survival outcomes represents a crucial step in treatment. The staging system in the 8th edition of the American Joint Committee on Cancer classifies patients based on tumor, node, and metastasis (TNM) staging. However, the survival rate 3-5 within the same stage cohort varies widely. It has been found that other independent prognostic factors including age, sex, histology, and treatment choices could significantly contribute to individualized predictions of survival. To improve the precision of lung cancer survival estimations, Cox proportional hazard models have gained popularity as a way of predicting outcomes. For example, the nomogram is a reliable tool that has demonstrated the ability to quantify risk by combining and clarifying significant clinical characteristics for clinical oncology. By drafting a concise chart of an outcome-risk predictive model, a nomogram derives the risk probability of a specific event, such as lung cancer–specific survival (LCSS). In various cancers, nomograms possess the ability to derive more precise risk predictions 8,9 when incorporated with TNM staging. However, these models have several limitations with respect to time-to-event prediction for the clinical management of patients with cancer, including the precise evaluation of overall survival and time to progression. Moreover, these models make linearity assumptions rather than perform nonlinear analyses that reflect real-world clinical 11 12 characteristics. Therefore, there is a need for better solutions that focus on nonlinear variables. Deep learning networks can learn the highly intricate and linear/nonlinear associations between prognostic clinical characteristics and an individual’s risk of death from LCSS. In application, these networks have even shown potential for providing individual recommendations based on the calculated risk. For example, by analyzing clinical data in the Surveillance, Epidemiology, and End Results (SEER) cancer registry, Bergquist et al assembled computerized methods including random forests, lasso regression, and neural networks to achieve 93% accuracy in predicting lung cancer stages. In another study, Corey et al developed a software package (Pythia) based on machine learning models that incorporated a patient’s age, sex, clinical baseline, race/ethnicity, and comorbidity history to determine the risk of postoperative complications or deaths. Matsuo et al also developed a deep learning network model that has demonstrated a higher C statistic than the traditional proportional hazard regression model (C statistic = 0.795 vs 0.784) for progression-free survival analysis. Furthermore, Katzman et al developed a novel deep learning method for survival analysis that uses a deep learning network to integrate Cox proportional hazards, which is referred to as the learning survival neural network (DeepSurv). The authors demonstrated that DeepSurv performed as well as published survival models and could be used to provide treatment recommendations for better survival outcomes. The present study design follows the American Joint Committee on Cancer criteria for model adoption and the transparent report of a deep learning architecture for individual prognosis and treatment recommendation. In this study, we first describe the performance of DeepSurv on real- life clinical data sets. Second, we elucidate how the DeepSurv model can learn complex associations between an individual’s characteristics and the outcome benefits of different treatments. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 2/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Methods Eligibility Criteria and Clinical Information All patients gave informed oral consent prior to data collection. After obtaining institutional review board approval from Shanghai Pulmonary Hospital, we selected patients from the SEER 18 Regs Research Data + Hurricane Katrina Impacted Louisiana Cases, Nov 2017 Sub, which includes clinical records on cancer occurrences in 18 areas of the United States and contains approximately 27.8% of the population. Clinical cases were included if the following criteria were met: pathologically confirmed primary stage I to IV NSCLC (only adenocarcinoma and squamous cell carcinoma) between January 2010 and December 2015 and the presence of 1 malignant primary lesion. From the SEER database (eTable 1 in the Supplement), we collected the baseline information of cases (sex, age, and marriage status), tumor characteristics (location, size, histologic grade, histologic type, TNM stage, SEER code (CS extension, CS mets at dx, regional nodes examined, regional nodes positive, lung– pleural/elastic layer invasion by H and E or elastic stain, lung–separate tumor nodules–ipsilateral lung, 19,20 lung–surgery to primary site]), and treatment details (surgical type). Patients were excluded if any of the included clinical characteristics status were unknown or missing. The outcome of interest in this study included LCSS according to specific codes provided by SEER (defined as the interval from surgery until death as a result of lung cancer). These patients were randomly divided into the training and validation cohort at a ratio of 8 to 2. To validate the DeepSurv model, an external test cohort was provided by the CHINA database. The cohort comprised 1182 patients with stage I to III NSCLC diagnosed between January 2009 and December 2013 in Shanghai Pulmonary Hospital, which are completely distinct from the patients in SEER database. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. Deep Learning Model Design In this study, DeepSurv was used to analyze patient-individual survival outcomes, which is a deep learning algorithm that can predict individual survival risk values (Figure 1). We use deep feed- forward neural network and the Cox proportional hazards model in survival analysis. The DeepSurv model contained a core hierarchical structure with fully connected feed-forward neural networks with a single output node to calculate the survival risks hθ(x ) of patients using the negative log-partial likelihood function. More details about the DeepSurv were described in the eMethods in the Supplement. Using the provided data set, we compared the performances of the TNM staging model and our deep learning model with respect to 2 tasks (LCSS predictions) with 3 different data sets (Figure 1). Data Analysis First, we developed a 6-layer neural network for predicting patient LCSS in the NSCLC training data set (n = 12 912). To validate the prediction performance, we used Harrell C statistic and calibration plots to evaluate the network discrimination and calibration in the NSCLC validation data set (n = 3228) and CHINA data set (n = 1182). Next, we trained a personalized treatment recommendation system using separately developed lobar and sublobar resection risk prediction models with a 3-layer neural network in the lobectomy (n = 10 766) and sublobectomy (n = 1444) training data sets. For each patient in the lobectomy and sublobectomy validation data set (SEER: n = 3064; CHINA: n = 1142), we chose the lower-risk value of the model’s treatment as the recommendation. Finally, we categorized the patients into 2 groups according to the consistency of the treatment received and recommended. For survival analysis, we used the Kaplan-Meier method to analyze LCSS between different groups and the log-rank test to compare survival curves. 6,17 An additional Cox proportional hazard regression model with nonneural network methods was performed following the simple backward-stepwise approach using all the variables included in JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 3/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival the DeepSurv model. It estimated the risk function h(x ) of the event occurring (LCSS) for patient i based on included features x using a linear function: h(Xi) =  x β (β = the coefficient of x ). i i i i Model Visualization We also developed a user-friendly interface to facilitate the survival predictions and treatment recommendations of the DeepSurv model. This interface consists of 3 views: (1) the user input view, (2) the survival prediction view, and (3) the treatment recommendation view. The user input view is designed to help users input all entries regarding patient characteristics using the XML schema constructed based on the features input into DeepSurv models. The user input view allows users to predict the survival probability and obtain a treatment recommendation based on specific patient information by clicking the predict and recommendation buttons, respectively. All SEER codes followed the SEER guideline. Statistical Analysis A 2-sided P value less than .05 was considered to be statistically significant. The Akaike information criterion was calculated to assess the risk of overfitting. The likelihood-based method was applied to the type I censoring design. All statistical analyses were performed with SPSS version 23 (IBM Corporation) software. The C statistic was performed by comparing C package with R statistical software (R Project for Statistical Computing), and the survival curves were plotted using GraphPad Prism 7 (GraphPad Software) software. Codes in our study are available online (https://github.com/thoraciclang/Deep_Lung). Analysis began January 2018 and ended June 2019. Figure 1. Diagram of the Study Procedure Training DeepSurv Test Linear Multilayer perception combination Prediction Survival SEER SEER CHINA h (x) n = 12 912 n = 3228 n = 1182 SEER lobectomy h (x)1 n = 10 766 h (x)1 SEER CHINA Surgery vs n = 3064 n = 1142 h (x)2 SEER sublobar resection h (x)2 n = 1444 Performance Stability Recommendation Visualization was benchmarked was assessed in the was tested by was applied using user-friendly against the TNM model validation cohort survival analysis graphic interface Deep learning networks were trained end to end on 3 data set groups. The training and node, and metastasis (TNM) models, assess their degrees of stability, formulate testing of these networks were all conducted on independent data sets. Four further recommendations, and finally, accomplish model visualization. SEER indicates experiments were conducted on the networks to test their performances against tumor, Surveillance, Epidemiology, and End Results. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 4/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Results Screening Process and Clinicopathology A total of 17 322 patients with NSCLC were included in the study. According to the screening criteria, a total of 16 140 patients diagnosed as having NSCLC from the SEER database were included (eFigure 1A in the Supplement). Table 1 shows the patients’ main baseline clinical characteristics (eTables 2 and 3 in the Supplement). The majority of patients were white (13 361 [82.8%]), and the median (interquartile range) age was 68 (61-74) years. The majority of tumors were stage I disease (9327 [57.8%]) and adenocarcinoma (11 037 [68.4%]). The median (interquartile range) follow-up time was 24 (10-43) months. There were 2893 patients (17.9%) who had events (deaths from NSCLC) during the follow-up time. There were 1182 patients diagnosed with NSCLC from CHINA database (eFigure 1B in the Supplement). There were 226 events (deaths from NSCLC) over a median (interquartile range) follow-up time of 63.3 (53-70) months. Table 1. Main Characteristics of Patients in the Whole Data Sets of Survival Analysis Data set, No. (%) Characteristic Training SEER (test 1) CHINA (test 2) Age at diagnosis, median (range), y 68 (28-95) 68 (19-92) 60 (30-87) Sex Female 6657 (51.6) 1639 (50.8) 642 (54.3) Male 6255 (48.4) 1589 (49.2) 540 (45.7) Histologic type Adenocarcinoma 8794 (68.1) 2243 (69.5) 948 (80.2) Squamous cell carcinoma 4118 (31.9) 985 (30.5) 234 (19.8) Marital status at diagnosis Unmarried 5304 (41.1) 1843 (57.1) 526 (44.5) Married 7608 (58.9) 1385 (42.9) 656 (55.5) T1a 563 (4.4) 139 (4.3) 128 (10.8) T1b 3156 (24.4) 804 (24.9) 396 (33.5) T1c 2342 (18.1) 641 (19.9) 346 (29.3) T2a 3258 (25.2) 791 (24.5) 208 (17.6) T2b 594 (4.6) 141 (4.4) 56 (4.7) T3 1994 (15.4) 445 (13.8) 40 (3.4) T4 1005 (7.8) 267 (8.3) 8 (0.7) N0 9712 (75.2) 2439 (75.6) 1030 (87.1) N1 1732 (13.4) 418 (12.9) 54 (4.6) N2 1422 (11) 356 (11) 98 (8.3) N3 46 (0.4) 15 (0.5) 0 M0 12 559 (97.3) 3132 (97) 1182 (100) M1a 143 (1.1) 41 (1.3) 0 M1b 202 (1.6) 52 (1.6) 0 M1c 8 (0.1) 3 (0.1) 0 LCCS Alive 10 581 (81.9) 2666 (82.6) 956 (80.9) Dead 2331 (18.1) 562 (17.4) 226 (19.1) Surgery to primary site Abbreviations: LCCS, lung cancer–specific survival; M, Pneumonectomy 613 (4.7) 132 (4.1) 40 (3.4) metastasis; N, node; SEER, Surveillance, Epidemiology, Lobectomy 10 766 (83.4) 2695 (83.5) 872 (73.8) and End Results cancer registry; T, tumor. Sublobar 1444 (11.2) 369 (11.4) 270 (22.8) Other detailed clinical characteristics can be found in None 89 (0.7) 32 (1.0) 0 eTables 2 and 3 in the Supplement. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 5/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Training Curves eFigure 2 in the Supplement demonstrated the training curves of networks in 3 submodels. The accuracy during the training course was indicated by validation and training lines. The curve was plotted to monitor the training course as the weights of the network were adjusted over each epoch, which represented the algorithm runs through the entire training and test data sets. After fine tuning, the change trend of loss and accuracy tended to become smoother and the algorithm maintained high accuracy on the validation set without significant overfitting. With a 500-epoch limit, we chose the model with the best performance on the test data set. Calibration and Validation of the Prognostic DeepSurv for LCSS in the Test Set We compared the TNM staging model to DeepSurv for LCSS in the test data sets (Table 2). The calibration plot indicated the calibration and how far the predictions of DeepSurv deviated from the actual event (Figure 2). In general, the actual outcomes in our databases of all patients with NSCLC for 3-year and 5-year LCSS were highly consistent with those predicted by the DeepSurv model, with most points falling almost directly on the 45° line. The DeepSurv model generated significantly better predictions than the TNM staging model (C statistic for TNM stage vs DeepSurv = 0.70; 95% CI, 0.681-0.731 vs 0.739; 95% CI, 0.713-0.764 [P < .001]). In the test group (CHINA data set), the DeepSurv model (C statistic = 0.742; 95% CI, 0.709-0.775) showed significantly better prediction than TNM model (C statistic = 0.706; 95% CI, 0.681-0.731; P < .001). High C statistic was observed for the results of the lobectomy and sublobar resection test data sets (Table 2). The feature component weightings in DeepSurv model are listed at eTable 4 in the Supplement. The Cox proportional hazard regression model (eTable 5 in the Supplement) was compared with the DeepSurv model for LCSS. The DeepSurv model had significantly better predictions compared with the Cox proportional hazard regression model (C statistic for Cox proportional hazard regression model vs DeepSurv model = 0.716; 95% CI, 0.705-0.727 vs 0.739; 95% CI, 0.713-0.764). The Akaike information criterion value of TNM stage model, Cox proportional hazard regression model, and DeepSurv model were 10741.89, 10307.08, and 10310.52, respectively. Treatment Recommender First, we plotted 2 Kaplan-Meier survival curves: the outcome of test cases whose actual treatments were the same as those recommended and those whose were not (eFigure 3 in the Supplement). The population that followed the recommendation experienced significantly better survival rates than those who did not (SEER: hazard ratio [HR], 2.99; 95% CI, 2.49-3.59; P < .001 vs CHINA: HR, 2.14; 95% CI, 1.65-2.77; P < .001). Furthermore, patients in the test data sets were classified into lobectomy and sublobar resection groups according to the received treatment. Consistent with prior analyses, LCSS favored lobectomy compared with sublobar resection in the subgroup with the lobectomy recommendation (SEER: HR, 1.79; 95% CI, 1.28-2.50; P = .001 vs CHINA: HR, 1.92; 95% CI, 1.30-2.83; P = .001). No significant distinction in survival results were observed for lobectomy and sublobar resection in the subgroup with the sublobar resection recommendation (SEER: HR, 0.65; 95% CI, 0.41-1.02; P = .06 vs CHINA: HR, 0.75; 95% CI, 0.44-1.77; P = .28). Table 2. Comparison of TNM Stage and DeepSurv Model for Survival Prediction in Test Data Sets LCCS outcome Model C statistic (95% CI) P value SEER TNM 0.706 (0.681-0.731) NA DeepSurv 0.739 (0.713-0.764) <.001 CHINA TNM 0.691 (0.659-0.724) NA Abbreviations: LCCS, lung cancer-specific survival; NA, DeepSurv 0.742 (0.709-0.775) <.001 not applicable; SEER, Surveillance, Epidemiology, and Treatment DeepSurv (lobectomy, SEER) 0.725 (0.698-0.751) NA End Results cancer registry; TNM, tumor, node, and DeepSurv (sublobar resection, SEER) 0.698 (0.672-0.725) NA metastasis. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 6/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Model Visualization In the prediction view, the system invokes a prediction model (Figure 3; Video), and the DeepSurv model is used to predict patients’ survival probability. The analysis results are visualized in a graphic view as a survival curve, which indicates the survival probability of the patient input over time. In the recommendation view, a recommendation model is adopted by the system, which can provide different patient survival probabilities for different treatments (lobectomy or sublobar resection) (Figure 3; Video). The survival curves of lobectomy and sublobar resection are also presented in a graphic view to facilitate visual comparison. Discussion The results of our pilot study proved that the deep learning network model (DeepSurv) performed better than conventional linear regression modeling (TNM staging model) in postoperative outcome prediction for patients with newly diagnosed NSCLC. Also, this model may serve as a useful analytical tool for treatment recommendation in patients with NSCLC, given its evidence of the significant prognostic benefits of following the treatment recommendation, which clearly outweigh those associated with not following the recommendation. Previous studies have reported a series of linear models to predict the survival of patients with 24-27 lung cancer. However, few risk factors have been selected to these models, which is significantly associated with the survival or recurrence. For example, Liang et al constructed a nomogram based on 6 factors. On the other hand, our Cox analysis demonstrated the contribution of 16 factors in the Figure 2. Calibration Plots for Lung Cancer–Specific Survival (LCCS) for the DeepSurv Model A B 3-Year survival in SEER data set 5-Year survival in SEER data set 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 Predicted 3-year LCCS Predicted 5-year LCCS C D 3-Year survival in CHINA data set 5-Year survival in CHINA data set 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 Three-year survival (A) and 5-year survival (B) of Surveillance, Epidemiology, and End Results (SEER) 0.2 0.2 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 data set and 3-year survival (C) and 5-year survival (D) Predicted 3-year LCCS Predicted 5-year LCCS of CHINA data set are shown. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 7/12 Actual 3-year LCCS Actual 3-year LCCS Actual 5-year LCCS Actual 5-year LCCS JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival DeepSurv model. Obviously, a more comprehensive analysis could be performed by a nonlinear deep learning model. After reviewing the most relevant advanced research, we found many studies to have already applied deep learning models in their analytical approaches to surgical oncology 13 28-30 research. However, most studies have focused on diagnostic application, such as radiographic 14,31-35 30,36-39 image automated quantification, digital histopathology image interpretation, or 11,40 biomarker analysis. Examples of published research using deep learning models for prognostic prediction in surgical oncology are rare, to our knowledge. In NSCLC research, the deep learning technique has been applied to digital histopathology image interpretation, driver mutation risk 40 33,41 detection, and image characteristics discrimination, but only a few studies have focused on postoperative outcome prediction or surgical recommendation, to our knowledge. As a new analytic tool, the deep learning network model will likely become more widely applied to support clinical decision-making. The performance of deep learning models in improving treatment outcomes is a key question and requires solid validation in the real world. In an analysis of 1194 patients with NSCLC, Hosny et al evaluated the prognostic signatures of quantitative imaging features, which were extracted using deep learning networks. Based on their study of the TNM stages of postoperative patients, the authors’ main finding was that deep learning networks significantly outperformed previous models. In our study, we selected a larger patient cohort with NSCLC of unselected consecutive cases including I to IV stages for model training and testing, which provide more solid results for interpretation. The advantages of the deep learning network model 42,43 for postoperative outcome prediction in surgical research can be summarized as follows. First, DeepSurv shows improved adaptability to variables with a nonlinear association, which includes real- world clinical factors. Unlike other models, deep learning algorithms can integrate the nonlinear risk functions associated with outcomes. Second, DeepSurv possesses flexibility in dealing with complex clinical factors. DeepSurv models cannot only automatically learn feature representations from uninterpreted clinical data but also analyze censored factors. Also, the predictions of the DeepSurv model have been proven to perform better in big data analysis. Owing to its ability to learn factor representation, the advantages of the DeepSurv model in dealing with both large factors and sample 44-46 size may play a big role in biomedical marker analyses. It is a surgeon’s duty to introduce clinical information to patients. To facilitate discussion of different potential surgical options, surgeons and patients need an informative tool that focuses on Figure 3. User-Friendly Interface of DeepSurv Model, Which Facilitates Survival Prediction and Treatment Recommendation Recommendation Predict Survival Patient Information 1.0 RX Summ-Scope Reg LN Sur Resection Female 0.9 Sex: (2003+): 0.8 0.7 Moderately Grade: CS extension (2004+): 0.6 0.5 White 15 Race Recode: 0.4 CS mets at dx (2004+): 0.3 Lung - Pleural/Elastic Layer 0.2 Marital Status Married Invasion (PL) by H and E or PL0 0.1 Elastic Stain: 0.0 Lung - Separate Tumor Nodules - 012 24 36 48 60 IA1 No seprate tumor Stage: lpsilateral Lung Time (months) Lung - Surgery to Primary Site T1a Lobectomy T Stage: (1988-2015): 1.0 Lung - Surgery to Other 0.9 N0 None N Stage: Regional/Distant Sites (1998+): 0.8 0.7 M0 60 M Stage: Age at diagnosis: 0.6 0.5 0.4 Upper lobe, lung Primary Site: 10 CS tumor size (2004+): 0.3 0.2 Regional nodes examined AD 20 Lobectomy Histologic: 0.1 (1998+): Sublobar 0.0 012 24 36 48 60 Regional nodes positive (1998+): Time (months) JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 8/12 Event Percent Event Percent JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival survival benefits. In real cases, the establishment of a user-friendly graphic interface based on a patient communication framework will be key to effectively conveying results and illustrating complex analyses, including prognostic prediction, treatment recommendation to patients and 44,47 family members, and improving the surgeons’ understanding of deep learning models. With its fast application and convenient operation, the user-friendly graphic interface example established in our study (Figure 3; Video) shows potential for use with any type of surgical care. To date, identifying patients who are appropriate for initial surgical management and conveying individualized prognostic analyses of postoperative outcomes has been an elusive goal. Instead, most published models are guided by patient characteristics to generate prognostic factors and are influenced by biases for different treatments. The DeepSurv model and its user-friendly graphic interface has the potential to address this clinical dilemma and better share individual outcomes following different surgical procedures. Limitations Since the innovation of deep learning models, many limitations have been recognized. First, deep learning network models are computationally expensive to train and validate. The process of predictions can be hard to interpret because the deep learning networks function much like black boxes, which make it difficult to determine how the network arrives at its decisions. We also recognize that single-clinical data sources have limited clinical characteristics compared with the automated quantification of radiographic images. In this study, we examined 127 features of 21 characteristics in the model. Some important factors including preoperative elements were neglected, which makes the recommendation system need more improvements and stay at feasibility trial status. Also, external validation is lacking in this study. Further study is needed to validate the advantages of deep learning networks in survival prediction. Conclusions To our knowledge, this study is the first to explore the performance of a deep learning network that integrates Cox proportional hazards (DeepSurv) in NSCLC and to obtain promising results in outcome prediction. In addition, we demonstrated DeepSurv’s potential to provide personalized treatment recommendations based on real clinical data. ARTICLE INFORMATION Accepted for Publication: March 20, 2020. Published: June 3, 2020. doi:10.1001/jamanetworkopen.2020.5842 Open Access: This is an open access article distributed under the terms of the CC-BY License.©2020SheYetal. JAMA Network Open. Corresponding Author: Chang Chen, MD, PhD (changchenc@tongji.edu.cn), and Yijiu Ren, MD (yjscott@hotmail.com), Department of Thoracic Surgery, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai 200443, China. Author Affiliations: Department of Thoracic Surgery, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, China (She, Wu, Deng, Zhang, Su, Jiang, Xie, Ren, Chen); College of Design and Innovation, Tongji University, Shanghai, China (Jin, Cao); Shanghai Key Laboratory of Tuberculosis, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, China (Liu); Computer Science, NYU Shanghai, Shanghai, China (Cao). Author Contributions: Dr Chen had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs She, Wu and Jin equally contributed to this work. Concept and design: Deng, Liu, Ren, Chen. Acquisition, analysis, or interpretation of data: She, Jin, Wu, Deng, Zhang, Su, Jiang, Xie, Cao, Ren, Chen. Drafting of the manuscript: Wu, Zhang, Jiang, Ren, Chen. JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 9/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival Critical revision of the manuscript for important intellectual content: She, Jin, Deng, Su, Liu, Xie, Cao, Ren, Chen. Statistical analysis: She, Jin, Wu, Deng, Zhang, Xie, Ren, Chen. Obtained funding: She, Xie, Ren, Chen. Administrative, technical, or material support: She, Jin, Su, Jiang, Xie, Ren, Chen. Supervision: Xie, Cao, Ren, Chen. Conflict of Interest Disclosures: Dr Chen reported grants from Shanghai Hospital Development Center during the conduct of the study. No other disclosures were reported. Funding/Support: This work was supported by projects from Shanghai Hospital Development Center (SHDC12012716), Shanghai Municipal Health Commission (2018ZHYL0102), Tongji University AI Program (12712150026), and Shanghai Pulmonary Hospital Innovation Program (FKCX1906). Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. REFERENCES 1. Chen W, Zheng R, Baade PD, et al. Cancer statistics in China, 2015. CA Cancer J Clin. 2016;66(2):115-132. doi:10. 3322/caac.21338 2. Asamura H, Chansky K, Crowley J, et al. The International Association for the Study of Lung Cancer Staging Project: proposals for the revision of the n descriptors in the forthcoming 8th edition of the TNM classification for lung cancer. J Thorac Oncol. 2015;10(12):1675-1684. doi:10.1097/JTO.0000000000000678 3. Chansky K, Sculier JP, Crowley JJ, Giroux D, Van Meerbeeck J, Goldstraw P; International Staging Committee and Participating Institutions. The International Association for the Study of Lung Cancer Staging Project: prognostic factors and pathologic TNM stage in surgically managed non-small cell lung cancer. J Thorac Oncol. 2009;4(7): 792-801. doi:10.1097/JTO.0b013e3181a7716e 4. Kawaguchi T, Takada M, Kubo A, et al. Performance status and smoking status are independent favorable prognostic factors for survival in non-small cell lung cancer: a comprehensive analysis of 26,957 patients with NSCLC. J Thorac Oncol. 2010;5(5):620-630. doi:10.1097/JTO.0b013e3181d2dcd9 5. Sculier JP, Chansky K, Crowley JJ, Van Meerbeeck J, Goldstraw P; International Staging Committee and Participating Institutions. The impact of additional prognostic factors on survival and their relationship with the anatomical extent of disease expressed by the 6th Edition of the TNM Classification of Malignant Tumors and the proposals for the 7th Edition. J Thorac Oncol. 2008;3(5):457-466. doi:10.1097/JTO.0b013e31816de2b8 6. Liang W, Zhang L, Jiang G, et al. Development and validation of a nomogram for predicting survival in patients with resected non-small-cell lung cancer. J Clin Oncol. 2015;33(8):861-869. doi:10.1200/JCO.2014.56.6661 7. Xie D, Marks R, Zhang M, et al. Nomograms predict overall survival for patients with small-cell lung cancer incorporating pretreatment peripheral blood markers. J Thorac Oncol. 2015;10(8):1213-1220. doi:10.1097/JTO. 8. Gold JS, Gönen M, Gutiérrez A, et al. Development and validation of a prognostic nomogram for recurrence-free survival after complete surgical resection of localised primary gastrointestinal stromal tumour: a retrospective analysis. Lancet Oncol. 2009;10(11):1045-1052. doi:10.1016/S1470-2045(09)70242-6 9. Callegaro D, Miceli R, Bonvalot S, et al. Development and external validation of two nomograms to predict overall survival and occurrence of distant metastases in adults after surgical resection of localised soft-tissue sarcomas of the extremities: a retrospective analysis. Lancet Oncol. 2016;17(5):671-680. doi:10.1016/S1470-2045 (16)00010-3 10. Randall RL, Cable MG. Nominal nomograms and marginal margins: what is the law of the line? Lancet Oncol. 2016;17(5):554-556. doi:10.1016/S1470-2045(16)00072-3 11. Li B, Cui Y, Diehn M, Li R. Development and validation of an individualized immune prognostic signature in early-stage nonsquamous non-small cell lung cancer. JAMA Oncol. 2017;3(11):1529-1537. doi:10.1001/jamaoncol. 2017.1609 12. Kopecky KE, Urbach D, Schwarze ML. Risk calculators and decision aids are not enough for shared decision making. JAMA Surg. 2019;154(1):3-4. 13. Hashimoto DA, Rosman G, Rus D, Meireles OR. Artificial intelligence in surgery: promises and perils. Ann Surg. 2018;268(1):70-76. doi:10.1097/SLA.0000000000002693 14. Hosny A, Parmar C, Coroller TP, et al. Deep learning for lung cancer prognostication: a retrospective multi- cohort radiomics study. PLoS Med. 2018;15(11):e1002711. doi:10.1371/journal.pmed.1002711 JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 10/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival 15. Bergquist SL, Brooks GA, Keating NL, Landrum MB, Rose S. Classifying lung cancer severity with ensemble machine learning in health care claims data. Proc Mach Learn Res. 2017;68:25-38. 16. Corey KM, Kashyap S, Lorenzi E, et al. Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): a retrospective, single-site study. PLoS Med. 2018;15(11):e1002701. doi:10.1371/journal.pmed.1002701 17. Matsuo K, Purushotham S, Jiang B, et al. Survival outcome prediction in cervical cancer: Cox models vs deep- learning model. Am J Obstet Gynecol. 2019;220(4):381.e1-381.e14. doi:10.1016/j.ajog.2018.12.030 18. Katzman JL, Shaham U, Cloninger A, Bates J, Jiang T, Kluger Y. DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Med Res Methodol. 2018; 18(1):24. doi:10.1186/s12874-018-0482-1 19. Shi S, Hua J, Liang C, et al. Proposed modification of the 8th edition of the AJCC staging system for pancreatic ductal adenocarcinoma. Ann Surg. 2019;269(5):944-950. doi:10.1097/SLA.0000000000002668 20. Wong SM, Freedman RA, Sagara Y, Aydogan F, Barry WT, Golshan M. Growing use of contralateral prophylactic mastectomy despite no improvement in long-term survival for invasive breast cancer. Ann Surg. 2017;265(3): 581-589. doi:10.1097/SLA.0000000000001698 21. Dai C, Ren Y, Xie D, et al. Does lymph node metastasis have a negative prognostic impact in patients with NSCLC and M1a disease? J Thorac Oncol. 2016;11(10):1745-1754. doi:10.1016/j.jtho.2016.06.030 22. Leung K-M, Elashoff RM, Afifi AA. Censoring issues in survival analysis. Annu Rev Public Health. 1997;18(1): 83-104. doi:10.1146/annurev.publhealth.18.1.83 23. GitHub. thoraciclang/Deep_Lung. Accessed May 11, 2020. https://github.com/thoraciclang/Deep_Lung 24. Wang Y, Qu X, Kam N-W, et al. An inflammation-related nomogram for predicting the survival of patients with non-small cell lung cancer after pulmonary lobectomy. BMC Cancer. 2018;18(1):692-692. doi:10.1186/s12885- 018-4513-4 25. Liang W, Zhang L, Jiang G, et al. Development and validation of a nomogram for predicting survival in patients with resected non-small-cell lung cancer. J Clin Oncol. 2015;33(8):861-869. doi:10.1200/JCO.2014.56.6661 26. Yap W-K, Shih M-C, Kuo C, et al. Development and validation of a nomogram for assessing survival in patients with metastatic lung cancer referred for radiotherapy for bone metastases. JAMA Netw Open. 2018;1(6): e183242-e183242. doi:10.1001/jamanetworkopen.2018.3242 27. Zhang Y, Zheng D, Xie J, et al. Development and validation of web-based nomograms to precisely predict conditional risk of site-specific recurrence for patients with completely resected non-small cell lung cancer: a multiinstitutional study. Chest. 2018;154(3):501-511. doi:10.1016/j.chest.2018.04.040 28. Carin L, Pencina MJ. On deep learning for medical image analysis. JAMA. 2018;320(11):1192-1193. doi:10.1001/ jama.2018.13316 29. Liang H, Tsui BY, Ni H, et al. Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. Nat Med. 2019;25(3):433-438. doi:10.1038/s41591-018-0335-9 30. Kather JN, Pearson AT, Halama N, et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med. 2019;25(7):1054-1056. doi:10.1038/s41591-019-0462-y 31. De Fauw J, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018;24(9):1342-1350. doi:10.1038/s41591-018-0107-6 32. Peake MD, Navani N, Baldwin DR. The continuum of screening and early detection, awareness and faster diagnosis of lung cancer. Thorax. 2018;73(12):1097-1098. doi:10.1136/thoraxjnl-2018-212189 33. Nam JG, Park S, Hwang EJ, et al. Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology. 2019;290(1):218-228. doi:10.1148/ radiol.2018180237 34. Hwang EJ, Park S, Jin KN, et al; DLAD Development and Evaluation Group. Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open. 2019;2(3):e191095. doi:10.1001/jamanetworkopen.2019.1095 35. Topalovic M, Das N, Burgel PR, et al; Pulmonary Function Study Investigators. Pulmonary Function Study Investigators. Artificial intelligence outperforms pulmonologists in the interpretation of pulmonary function tests. Eur Respir J. 2019;53(4):1801660. doi:10.1183/13993003.01660-2018 36. Ren J, Karagoz K, Gatza ML, et al. Recurrence analysis on prostate cancer patients with Gleason score 7 using integrated histopathology whole-slide images and genomic data through deep neural networks. J Med Imaging (Bellingham). 2018;5(4):047501. doi:10.1117/1.JMI.5.4.047501 JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 11/12 JAMA Network Open | Oncology Development and Validation of a Deep Learning Model for Non–Small Cell Lung Cancer Survival 37. Campanella G, Hanna MG, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med. 2019;25(8):1301-1309. doi:10.1038/s41591-019-0508-1 38. Gertych A, Swiderska-Chadaj Z, Ma Z, et al. Convolutional neural networks can accurately distinguish four histologic growth patterns of lung adenocarcinoma in digital slides. Sci Rep. 2019;9(1):1483. doi:10.1038/s41598- 018-37638-9 39. Steiner DF, MacDonald R, Liu Y, et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol. 2018;42(12):1636-1646. doi:10.1097/PAS. 40. Riesselman AJ, Ingraham JB, Marks DS. Deep generative models of genetic variation capture the effects of mutations. Nat Methods. 2018;15(10):816-822. doi:10.1038/s41592-018-0138-4 41. Tanaka N, Kanatani S, Tomer R, et al. Whole-tissue biopsy phenotyping of three-dimensional tumours reveals patterns of cancer heterogeneity. Nat Biomed Eng. 2017;1(10):796-806. doi:10.1038/s41551-017-0139-0 42. Yasaka K, Abe O. Deep learning and artificial intelligence in radiology: current applications and future directions. PLoS Med. 2018;15(11):e1002707. doi:10.1371/journal.pmed.1002707 43. Smith CC, Chai S, Washington AR, et al. Machine-learning prediction of tumor antigen immunogenicity in the selection of therapeutic epitopes. Cancer Immunol Res. 2019;7(10):1591-1604. doi:10.1158/2326-6066.CIR- 19-0155 44. Wang F, Casalino LP, Khullar D. Deep learning in medicine-promise, progress, and challenges. JAMA Intern Med. 2019;179(3):293-294. 45. Shaffie A, Soliman A, Fraiwan L, et al. A generalized deep learning-based diagnostic system for early diagnosis of various types of pulmonary nodules. Technol Cancer Res Treat. 2018;17:1533033818798800. doi:10.1177/ 46. Kantz ED, Tiwari S, Watrous JD, Cheng S, Jain M. Deep neural networks for classification of LC-MS spectral peaks. Anal Chem. 2019;91(19):12407-12413. doi:10.1021/acs.analchem.9b02983 47. Simon G, DiNardo CD, Takahashi K, et al. Applying Artificial Intelligence to Address the Knowledge Gaps in Cancer Care. Oncologist. 2019;24(6):772-782. 48. Coiera E. On algorithms, machines, and medicine. Lancet Oncol. 2019;20(2):166-167. doi:10.1016/S1470-2045 (18)30835-0 SUPPLEMENT. eMethods. eTable 1. All Clinical Features Integrated in the Model eTable 2. Characteristics of Patients in the Training Dataset of Survival Analysis eTable 3. Characteristics of Patients in the Test Dataset of Survival Analysis eTable 4. Feature Component Weightings in the DeepSurv Model eTable 5. Survival Predictors in Cox PH Model eFigure 1. Flow chart of datasets construction. (A) SEER dataset, (B) CHINA dataset eFigure 2. Training curves of networks in the survival dataset of SEER database (A), lobectomy dataset (B), and sublobar resection dataset (C) eFigure 3. Lung Cancer–Specific Survival Recommendation Comparisons of SEER Data set (A), SEER Lobectomy Test Data set (B), and SEER Sublobar Resection Test Data set (C); Lung Cancer–Specific Survival Recommendation Comparisons of CHINA Data set (D), CHINA Lobectomy Test Data set (E), and CHINA sublobar Resection Test Data set (F) JAMA Network Open. 2020;3(6):e205842. doi:10.1001/jamanetworkopen.2020.5842 (Reprinted) June 3, 2020 12/12 eMethods. eTable 1. All Clinical Features Integrated in the Model Characteristics Features Sex Female Male Main Lower Overlapping Lung, bronchus/Carin Upper lobe, Primary Site Middle lobe, lung lobe, lesion of NOS/Bronch a/Hilum/Bronch lung lung lung us, NOS us intermedius Histologic Type AD SD Grade Moderately Poor undifine well Stage IA1 IA2 IA3 IB IIA IIB IIIA IIIB IIIC IVA IVB T stage T1a T1b T1c T2a T2b T3 T4 N stage N0 N1 N2 N3 M stage M0 M1a M1b M1c RX Summ--Scope System Reg LN Sur None Biopsy Resection resection (2003+) Microscopic focus or foci CS tumor size Diffuse (entire 1-988mm only and no (2004+) lobe) size of focus given Blood Tumor vessel(s) of/involvi Direct , major + ng main extensio Direct stem n extensio bronchus Extensio Tumor to:Brachi Superior n less than n proven al sulcus to:Brachi 2.0 cm to:Pleura by plexus, tumor al from , visceral presence inferior WITH plexus, carina + or NOS of Atelectasis/obs Extensio branches encasem inferior Atelectas (WITHO Tumor malignan tructive n Stated or NOS, ent of branches Superficial tumor Extensio Invasion Tumor is/obstru UT of/involvi t cells in Stated pneumonitis to:Pleura Stated Stated as T2 Atelectas Stated from subclavi Multiple or NOS, Pleural Stated Stated of any size with n from Direct Extension to but of of/involvi ctive pleural ng main sputum as T1a Stated as Tumor that extends to , visceral as T2a as T2b [NOS] is/obstru as T3 superior an masses/ tumor as T4 as T4 from invasive Stated as other tumor not into pleura, pleura, ng main pneumon effusion) stem Parietal Aorta + Adjacent Malignan Further or with no T1b with no involving the hilar region Invasion or NOS with no with no with no ctive with no Invasion sulcus/C vessels separate superior Heart/Vis foci Adjacent with no with no In situ, Tumor component limited T1[NOS] with parts of invasion Tumor Tumor including invasion including Pulmona stem itis that Pulmona bronchus pericardi Tumor Blood Malignan Heart/Vis Aorta + Vertebra Inferior rib + t contiguo bronchial CS extension other other main Localize but does not of (WITHO other other other pneumon other of hest OR tumor sulcus/C ceral Adjacent separate rib other other to bronchial wall, lung to invasion bronchus Aorta intraepithelial, confined to no other into an confined confined of elastic ry extends ry less than um or confined vessel(s) t pleural ceral Adjacent vena Blood pericardi us washing (2004+) informati information stem d, NOS involve the pleura, UT informati informati informati itis informati phrenic (thoracic WITH nodule(s hest pericardi rib from +other informati informati noninvasive one lung with or without information main adjacent to hilus to carina layer/BUT not through ligament less than to the ligament 2.0 cm pericardi to carina , major effusion pericardi rib foramina cava vessel(s) al extensio s but not on on on bronchus entire lung Or NOS pleural on on on on on on involving on on nerve ) unequivo ) in the (thoracic um direct extention on on on on proximal extension on extension stem ipsilatera through the elastic the 2.0 cm hilar + Tumor from um, NOS um , major effusion n visualize extensio extension , NOS atelectasis/obs effusion) extensio extensio size or entire extensio wall/Diap cal SAME pleural extensio extensio to the main stem bronchus l lobe layer elastic from region of/involvi carina + d by n tructive Pulmona n n extensio lung n hragm/P involvem lobe wall/Diap invasion n n bronchus , NOS layer carina but does ng main Invasion imaging pneumonitis, ry n ancoast ent of hragm/P not stem of pleura or NOS ligament tumor superior ancoast involve bronchus bronchos (superior branches tumor the less than copy; sulcus of (superior entire 2.0 cm "occult" syndrom brachial sulcus lung Or from carcinom e), plexus syndrom atelectas carina a NOS/Par e), is/obstru ietal NOS/Par ctive pleura ietal pneumon pleura/S itis, NOS uperior Extensio to:Skelet Abdomin Extensio al al n muscle/S organs+ to:Contra ternum/S Distant Contralat lateral kin of metastas eral lung/Con Extensio chest Extensio es + lung/Con tralateral n to +Contral n Distant tralateral Distant Malignan main Pleural contralat ateral to:Contra node(s) Malignant main metastas Distant Stated t stem tumor eral lung Stated lung/Con lateral + Stated pleural Separate Distant Extensio stem Distant is plus metastas as M1 Malignan Malignant pericardi bronchus foci or plus as M1a Distant tralateral lung/Con Contralat as M1b effusion, tumor lymph n bronchus metastas Distant Distant distant es plus [NOS] Distant lymph t pleural pleural Malignan al /Separat nodules pleural with no lymph main tralateral eral with no ipsilateral Distant lymph Distant lymph nodule(s nodes to:Skelet /Separat is plus metastas metastas lymph distant with no node(s), Malignant pleural effusion, effusion, t effusion e tumor on the or other nodes stem main lung/Con other CS mets at dx No distant and node(s), nodes plus pleural ) in plus al Abdomin e tumor pleural is plus is plus nodes lymph other including effusion, ipsilateral contralat unknown if pericardi plus nodule(s ipsilatera pericardi informati plus bronchus stem tralateral informati (2004+) metastasis contralateral including or pericardial different extensio muscle/S al organs nodule(s or pleural distant plus nodes informati cervical or same lung eral or ipsilateral or al contralat ) in l lung al on on pleural /Separat bronchus main on on lungs cervical nodes effusion lobe, n to ternum/S ) in pericardi tumor lymph pleural plus on on nodes other contralateral effusion eral or contralat separate effusion distant tumor e tumor /Separat stem distant (Bilateral same contralat kin of contralat al foci node(s) or pleural distant lung lung bilateral eral from or metastas foci nodule(s e tumor bronchus metastas pleural lung eral lung chest eral effusion pericardi tumor metastas pleural lung/Ple direct separate is ) in nodule(s /Separat is effusion) lung/Ple al foci is effusion ural invasion pleural contralat ) in e tumor ural effusion tumor tumor eral contralat nodule(s tumor foci or foci lung/Ple eral lung ) in foci or nodules ural contralat nodules on tumor eral lung on contralat foci or contralat eral lung nodules eral lung on contralat eral lung Regional nodes 1-100 examined (1988+) Regional nodes 1-100 positive (1988+) American Asian or African Race recode White Indian/Alaska Pacific Americans Native Islander Age at diagnosis 1-120 Marital status at diagnosis married un-married married=1 un- married=0 Lung - Pleural/Elastic Layer Invasion PL0 PL1 PL2 PL3 (PL) by H and E or Elastic Stain Separate tumor Separate nodules, Separate tumor Lung - Separate No separate tumor nodules ipsilatera nodules in Tumor Nodules - tumor nodules in ipsilateral l lung, ipsilateral lung, Ipsilateral Lung noted lung, different same same lobe lobe and different lobe Lung - Surgery to Primary Site Peumonectomy Lobectomy Sublobar None (1988-2015) Lung - Surgery to Surgery To Other Surgery To None Distant Site Regional/Distant Regional Site Or Nodes Sites (1998+) eTable 2. Characteristics of Patients in the Training Dataset of Survival Analysis IA1 IA2 IA3 IB IIA IIB IIIA IIIB IIIC IVA IVB total Stage median range median range median range median range mediarange mediarange median range median range median range medianrange median range median range Age at diagnosis 67 36-90 67 29-90 69 31-92 69 32-92 69 35-86 67 30-95 67 29-90 66 34-89 64 46-85 65 28-84 61 50-78 68 28-95 count % count % count % count % count % count % count % count % count % count % count % count % Sex Female 324 62.5 1585 57.5 1079 56.7 1158 51.3 180 45 1025 44.9 908 47 213 42.9 5 29.4 175 50.7 5 62.5 6657 51.6 Male 194 37.5 1172 42.5 823 43.3 1098 48.7 220 55 1258 55.1 1022 53 283 57.1 12 70.6 170 49.3 3 37.5 6255 48.4 Histologic Type 2 AD 410 79.2 2110 76.5 1347 70.8 1492 66.1 223 55.8 1384 60.6 1245 64.5 307 61.9 10 58.8 260 75.4 6 75 8794 68.1 SD 108 20.8 647 23.5 555 29.2 764 33.9 177 44.3 899 39.4 685 35.5 189 38.1 7 41.2 85 24.6 2 25 4118 31.9 Marital status at diagnosis un-married 203 39.2 1149 41.7 805 42.3 970 43 163 40.8 931 40.8 753 39 191 38.5 4 23.5 133 38.6 2 25 5304 41.1 married 315 60.8 1608 58.3 1097 57.7 1286 57 237 59.3 1352 59.2 1177 61 305 61.5 13 76.5 212 61.4 6 75 7608 58.9 T1a 518 100 0 0 0 0 0 0 0 0 22 1 21 1.1 0 0 0 0 2 0.6 0 0 563 4.4 T1b 0 0 2757 100 0 0 0 0 0 0 219 9.6 146 7.6 5 1 0 0 29 8.4 0 0 3156 24.4 T1c 0 0 0 0 1902 100 0 0 0 0 217 9.5 182 9.4 4 0.8 0 0 36 10.4 1 12.5 2342 18.1 T2a 0 0 0 0 0 0 2256 100 0 0 499 21.9 425 22 4 0.8 0 0 73 21.2 1 12.5 3258 25.2 T2b 0 0 0 0 0 0 0 0 400 100 111 4.9 71 3.7 1 0.2 0 0 11 3.2 0 0 594 4.6 T3 0 0 0 0 0 0 0 0 0 0 1214 53.2 381 19.7 284 57.3 11 64.7 102 29.6 2 25 1994 15.4 T4 0 0 0 0 0 0 0 0 0 0 1 0 704 36.5 198 39.9 6 35.3 92 26.7 4 50 1005 7.8 N0 518 100 2757 100 1902 100 2256 100 400 100 1216 53.3 484 25.1 0 0 0 0 177 51.3 2 25 9712 75.2 N1 0 0 0 0 0 0 0 0 0 0 1067 46.7 601 31.1 0 0 0 0 63 18.3 1 12.5 1732 13.4 N2 0 0 0 0 0 0 0 0 0 0 0 0 845 43.8 482 97.2 0 0 92 26.7 3 37.5 1422 11 N3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14 2.8 17 100 13 3.8 2 25 46 0.4 M0 518 100 2757 100 1902 100 2256 100 400 100 2283 100 1930 100 496 100 17 100 0 0 0 0 12559 97.3 M1a 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 143 41.4 0 0 143 1.1 M1b 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 202 58.6 0 0 202 1.6 M1c 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 100 8 0.1 LCCS Alive 496 95.8 2577 93.5 1701 89.4 1950 86.4 340 85 1758 77 1303 67.5 279 56.3 7 41.2 169 49 1 12.5 10581 81.9 dead 22 4.2 180 6.5 201 10.6 306 13.6 60 15 525 23 627 32.5 217 43.8 10 58.8 176 51 7 87.5 2331 18.1 Pleural/Elastic Layer Invasion PL0 518 100 2757 100 1902 100 1369 60.7 399 99.8 1715 75.1 1450 75.1 327 65.9 14 82.4 239 69.3 8 100 10698 82.9 PL1 0 0 0 0 0 0 489 21.7 1 0.3 221 9.7 181 9.4 67 13.5 1 5.9 37 10.7 0 0 997 7.7 PL2 0 0 0 0 0 0 398 17.6 0 0 157 6.9 194 10.1 57 11.5 1 5.9 43 12.5 0 0 850 6.6 PL3 0 0 0 0 0 0 0 0 0 0 190 8.3 105 5.4 45 9.1 1 5.9 26 7.5 0 0 367 2.8 Separate Tumor Nodules 1 518 100 2757 100 1902 100 2255 100 400 100 1903 83.4 1617 83.8 348 70.2 14 82.4 241 69.9 3 37.5 11958 92.6 2 0 0 0 0 0 0 0 0 0 0 0 0 158 8.2 47 9.5 0 0 42 12.2 3 37.5 250 1.9 3 0 0 0 0 0 0 1 0 0 0 380 16.6 126 6.5 91 18.3 2 11.8 45 13 1 12.5 646 5 4 0 0 0 0 0 0 0 0 0 0 0 0 29 1.5 10 2 1 5.9 17 4.9 1 12.5 58 0.4 Surgery to Other Regional/Distant Sites None 515 99.4 2748 99.7 1892 99.5 2236 99.1 399 99.8 2252 98.6 1885 97.7 480 96.8 14 94.1 275 79.7 7 87.5 12705 98.4 Distant 0 0 4 0.1 4 0.2 7 0.3 0 0 11 0.5 11 0.6 4 0.8 0 0 64 18.6 1 12.5 106 0.8 Regional 3 0.6 5 0.2 6 0.3 13 0.6 1 0.3 20 0.9 34 1.8 12 2.4 1 5.9 6 1.7 0 0 101 0.8 eTable 3. Characteristics of Patients in the Test Dataset of Survival Analysis Stage IA1 IA2 IA3 IB IIA IIB IIIA IIIB IIIC IVA IVB total median range median range median range median range median range median range median range median range median range median range median range median range Age at diagnosis 68 34-86 67 35-90 69 36-92 69 29-90 68 40-87 68 42-91 67 39-88 66 19-87 63 36-68 65 40-83 67 59-69 68 19-92 count % count % count % count % count % count % count % count % count % count % count % count % Sex 3 3228 Female 82 66.1 407 57.2 267 52 278 50.9 35 35.4 267 50.1 203 41.7 51 44 2 66.7 45 48.4 2 66.7 1639 50.8 Male 42 33.9 304 42.8 246 48 268 49.1 64 64.6 266 49.9 284 58.3 65 56 1 33 48 51.6 1 33.3 1589 49.2 Histologic Type 2 AD 99 79.8 541 76.1 369 71.9 380 69.6 48 48.5 346 64.9 319 65.5 73 62.9 2 66.7 64 68.8 2 66.7 2243 69.5 SD 25 20.2 107 23.9 144 28.1 166 30.4 51 51.5 187 35.1 168 34.5 43 37.1 1 33.3 29 31.2 1 33.3 985 30.5 Marital status at diagnosis married 59 47.6 308 43.3 294 57.3 305 55.9 40 40.4 295 55.3 295 60.6 69 59.5 1 33.3 56 60.2 1 33.3 1843 57.1 un-married 65 52.4 403 56.7 219 42.7 241 44.1 59 59.6 238 44.7 192 39.4 47 40.5 2 66.7 37 39.8 2 66.7 1385 42.9 T1a 124 100 0 0 0 0 0 0 0 0 7 1.3 7 1.4 0 0 0 0 4 1.1 0 0 139 4.3 T1b 0 0 711 100 0 0 0 0 0 0 39 7.3 43 8.8 2 1.7 0 0 9 9.7 0 0 804 24.9 T1c 0 0 0 0 513 100 0 0 0 0 65 12.2 52 10.7 2 1.7 0 0 9 9.7 0 0 641 19.9 T2a 0 0 0 0 0 0 546 100 0 0 126 23.6 97 19.9 2 1.7 0 0 20 21.5 0 0 791 24.5 T2b 0 0 0 0 0 0 0 0 99 100 18 3.4 19 3.9 0 0 0 0 5 5.4 0 0 141 4.4 T3 0 0 0 0 0 0 0 0 0 0 278 52.2 76 15.6 67 57.8 1 33.3 22 23.7 1 33.3 445 13.8 T4 0 0 0 0 0 0 0 0 0 0 0 0 193 39.6 43 37.1 2 66.7 27 29 2 66.7 267 8.3 N0 124 100 711 100 513 100 546 100 99 100 278 52.2 120 24.6 0 0 0 0 47 50.5 1 33.3 2439 75.6 N1 0 0 0 0 0 0 0 0 0 0 255 47.8 149 30.6 0 0 0 0 13 14 1 33.3 418 12.9 N2 0 0 0 0 0 0 0 0 0 0 0 0 218 44.8 110 94.8 0 0 27 29 1 33.3 356 11 N3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 5.2 4 100 6 6.5 0 0 15 0.5 M0 124 100 711 100 513 100 546 100 99 100 533 100 487 100 116 100 3 100 0 0 0 0 3132 97 M1a 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 41 44.1 0 0 41 1.3 M1b 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 52 55.9 0 0 52 1.6 M1c 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 100 3 0.1 LCCS Alive 121 97.6 657 92.4 470 91.6 478 87.5 91 91.9 413 77.5 321 65.9 66 56.9 1 33.3 47 50.5 1 33.3 2666 82.6 dead 3 2.4 54 7.6 43 8.4 68 12.5 8 8.1 120 22.5 166 34.1 50 43.1 2 66.7 46 49.5 2 66.7 562 17.4 Pleural/Elastic Layer Invasion PL0 124 100 711 100 513 100 315 57.7 99 100 405 76 360 73.9 80 69 2 66.7 66 71 1 33.3 2676 82.9 PL1 0 0 0 0 0 0 133 24.4 0 0 47 8.8 50 10.3 10 8.6 0 0 8 8.6 1 33.3 249 7.7 PL2 0 0 0 0 0 0 98 17.9 0 0 37 6.9 52 10.7 17 14.7 1 33.3 15 16.1 1 33.3 221 6.8 PL3 0 0 0 0 0 0 0 0 0 0 44 8.3 25 5.1 9 7.8 0 0 4 4.3 0 0 82 2.5 Separate Tumor Nodules 1 124 100 711 100 513 100 546 100 99 100 433 83.1 410 84.2 82 70.7 3 100 69 74.2 2 66.7 3002 93 2 0 0 0 0 0 0 0 0 0 0 0 0 36 7.4 10 8.6 0 0 10 10.8 1 33.3 57 1.8 3 0 0 0 0 0 0 0 0 0 0 90 16.9 28 5.7 23 19.8 0 0 10 10.8 0 0 151 4.7 4 0 0 0 0 0 0 0 0 0 0 0 0 13 2.7 1 0.9 0 0 4 4.3 0 0 18 0.6 Surgery to Other Regional/Distant Sites None 124 100 711 100 513 100 545 99.8 99 100 527 98.9 478 98.2 115 99.1 3 100 74 79.6 2 66.7 3181 98.5 Distant 0 0 0 0 0 0 0 0 0 0 0 0 2 0.4 0 0 0 0 17 18.3 1 33.3 24 0.7 Regional 0 0 0 0 0 0 1 0.2 0 0 6 1.1 7 1.4 1 0.9 0 0 2 2.2 0 0 23 0.7 eTable 4. Feature Component Weightings in the DeepSurv Model Features Weight Features Weight Features Weight Age at diagnosis 0.5722479 CS extension (2004+)=540 0.42365953 Stage=3 -0.0717 CS tumor size (2004+) 0.6724694 CS extension (2004+)=550 0.08433475 Stage=4 -0.07168 Regional nodes examined (1988+) -0.4995487 CS extension (2004+)=560 0.13039102 Stage=5 -0.01987 Regional nodes positive (1988+) 0.7574372 CS extension (2004+)=570 0.06113536 Stage=6 0.041379 Sex=1 -0.062900014 CS extension (2004+)=590 0.22531554 Stage=7 0.101898 Sex=2 0.1671767 CS extension (2004+)=600 0.12043501 Stage=8 0.093093 Histologic Type ICD-O-3=0 0.055728845 CS extension (2004+)=610 -0.90538454 Stage=9 0.104782 Histologic Type ICD-O-3=1 -0.048216447 CS extension (2004+)=680 -0.824522 Stage=10 0.034815 Histologic Type ICD-O-3=2 0.015860233 CS extension (2004+)=700 0.16996412 Stage=11 0.51159 Histologic Type ICD-O-3=3 -0.08446639 CS extension (2004+)=705 -0.9847599 T8=1 0.209211 Histologic Type ICD-O-3=4 0.0417398 CS extension (2004+)=710 0.3103478 T8=2 0.038959 Histologic Type ICD-O-3=5 -0.00887268 CS extension (2004+)=730 -0.20042646 T8=3 0.056467 Histologic Type ICD-O-3=6 0.025923487 CS extension (2004+)=740 0.81883913 T8=4 0.049242 Histologic Type 2=1 -0.028282069 CS extension (2004+)=745 0.40200815 T8=5 0.06238 Histologic Type 2=2 0.020090567 CS extension (2004+)=750 0.16024342 T8=6 -0.09599 Grade=1 -0.002670259 CS extension (2004+)=770 -0.86313397 T8=7 -0.12861 Grade=2 0.063437365 CS extension (2004+)=785 -0.17053518 N8=1 -0.06939 Grade=3 0.15810749 CS mets at dx (2004+)=0 -0.07108595 N8=2 0.052847 Grade=4 -0.124901354 CS mets at dx (2004+)=15 0.14313275 N8=3 0.070122 RX Summ--Scope Reg LN Sur (2003+) -0.05786139 CS mets at dx (2004+)=16 0.08157881 N8=4 0.202186 RX Summ--Scope Reg LN Sur (2003+) 0.029179208 CS mets at dx (2004+)=17 -0.34559816 M8=1 -0.21266 RX Summ--Scope Reg LN Sur (2003+) 0.020770853 CS mets at dx (2004+)=18 0.059344094 M8=2 -0.02519 RX Summ--Scope Reg LN Sur (2003+) -0.03520994 CS mets at dx (2004+)=20 -0.12556794 M8=3 0.030704 CS extension (2004+)=100 -0.0725175 CS mets at dx (2004+)=21 0.99101025 M8=4 0.576268 CS extension (2004+)=110 0.04731824 CS mets at dx (2004+)=23 -0.19575515 Marital status at diagnosis=0 0.034368 CS extension (2004+)=115 0.026038347 CS mets at dx (2004+)=24 -0.037997384 Marital status at diagnosis=1 -0.10559 CS extension (2004+)=120 0.004864246 CS mets at dx (2004+)=25 -0.76219594 Lung - Pleural/Elastic Layer Invasion (PL) by H and E or Elastic Stain=0 0.001433 CS extension (2004+)=125 -0.09474413 CS mets at dx (2004+)=26 0.90065193 Lung - Pleural/Elastic Layer Invasion (PL) by H and E or Elastic Stain=1 0.034629 CS extension (2004+)=200 0.021685144 CS mets at dx (2004+)=30 -0.18796362 Lung - Pleural/Elastic Layer Invasion (PL) by H and E or Elastic Stain=2 0.001899 CS extension (2004+)=210 -0.16040157 CS mets at dx (2004+)=32 0.6542427 Lung - Pleural/Elastic Layer Invasion (PL) by H and E or Elastic Stain=3 0.164463 CS extension (2004+)=220 -0.10082796 CS mets at dx (2004+)=33 -1.1801782 Lung - Separate Tumor Nodules - Ipsilateral Lung=1 -0.03875 CS extension (2004+)=230 -0.21979994 CS mets at dx (2004+)=36 0.88033414 Lung - Separate Tumor Nodules - Ipsilateral Lung=2 0.063924 CS extension (2004+)=300 -0.033153117 CS mets at dx (2004+)=37 -0.1698591 Lung - Separate Tumor Nodules - Ipsilateral Lung=3 0.042076 CS extension (2004+)=400 -0.029358057 CS mets at dx (2004+)=40 0.14898834 Lung - Separate Tumor Nodules - Ipsilateral Lung=4 0.150139 CS extension (2004+)=410 -0.102100626 CS mets at dx (2004+)=41 0.5691616 Lung - Surgery to Primary Site (1988-2015)=1 0.005233 CS extension (2004+)=420 -0.07669448 CS mets at dx (2004+)=42 0.032155376 Lung - Surgery to Primary Site (1988-2015)=2 -0.13327 CS extension (2004+)=430 0.057814617 CS mets at dx (2004+)=43 0.030705813 Lung - Surgery to Primary Site (1988-2015)=3 -0.03041 CS extension (2004+)=440 0.22026922 CS mets at dx (2004+)=51 0.6830787 Lung - Surgery to Primary Site (1988-2015)=4 0.141988 CS extension (2004+)=455 -0.01990803 CS mets at dx (2004+)=52 1 Lung - Surgery to Other Regional/Distant Sites (1998+)=1 -0.04313 CS extension (2004+)=460 -0.061044298 CS mets at dx (2004+)=53 0.6274554 Lung - Surgery to Other Regional/Distant Sites (1998+)=2 0.117256 CS extension (2004+)=465 0.56912345 CS mets at dx (2004+)=70 -0.58639836 Lung - Surgery to Other Regional/Distant Sites (1998+)=3 0.06074 CS extension (2004+)=500 -0.10761972 Stage=1 -0.4348911 CS extension (2004+)=520 0.20671241 Stage=2 -0.15681928 eTable 5. Survival Predictors in Cox PH Model eFigure 1. Flow chart of datasets construction. (A) SEER dataset, (B) CHINA dataset eFigure 2. Training curves of networks in the survival dataset of SEER database (A), lobectomy dataset (B), and sublobar resection dataset (C). The red and purple curves indicate loss of the training and test datasets, respectively; the blue and yellow curves indicate the accuracy of the training and test datasets, respectively. eFigure 3. Lung cancer specific survival recommendation comparisons of SEER dataset (A), SEER lobectomy test dataset (B), and SEER sublobar resection test dataset (C); Lung cancer specific survival recommendation comparisons of CHINA dataset (D), CHINA lobectomy test dataset (E), and CHINA sublobar resection test dataset (F).

Journal

JAMA Network OpenAmerican Medical Association

Published: Jun 3, 2020

References