SCIEnTIFIC REPoRTS | (2018) 8:3395 | DOI:10.1038/s41598-018-21758-3
Deep learning based tissue analysis
predicts outcome in colorectal
, Nina Linder
, Riku Turkki
, Stig Nordling
, Panu E. Kovanen
, Margarita Walliander
, Mikael Lundin
, Caj Haglund
& Johan Lundin
Image-based machine learning and deep learning in particular has recently shown expert-level accuracy
in medical image classication. In this study, we combine convolutional and recurrent architectures to
train a deep network to predict colorectal cancer outcome based on images of tumour tissue samples.
The novelty of our approach is that we directly predict patient outcome, without any intermediate
tissue classication. We evaluate a set of digitized haematoxylin-eosin-stained tumour tissue
microarray (TMA) samples from 420 colorectal cancer patients with clinicopathological and outcome
data available. The results show that deep learning-based outcome prediction with only small tissue
areas as input outperforms (hazard ratio 2.3; CI 95% 1.79–3.03; AUC 0.69) visual histological assessment
performed by human experts on both TMA spot (HR 1.67; CI 95% 1.28–2.19; AUC 0.58) and whole-slide
level (HR 1.65; CI 95% 1.30–2.15; AUC 0.57) in the stratication into low- and high-risk patients. Our
results suggest that state-of-the-art deep learning techniques can extract more prognostic information
from the tissue morphology of colorectal cancer than an experienced human observer.
Reincarnation of articial neural networks in the form of deep learning
has improved the accuracy of several
pattern recognition tasks, such as classication of objects, scenes and various other entities in digital images. In
a biomedical context promising results have been achieved in image-based diagnostics ranging from ophthal-
to diagnostic pathology
. Within digital pathology, quantication and classication of digitized tissue
samples by supervised deep learning has shown good results even for tasks previously considered too challenging
to be accomplished with conventional image analysis methods
Oen, the purpose of many tasks in digital pathology, such as counting mitoses
, quantifying tumour inl-
trating immune cells
, assessing the grade of tumour dierentiation
or characterization of specic tissue
aim to ultimately predict patient outcome
. erefore, an interesting question is whether these
intermediate proxies for outcome could be bypassed and the novel machine learning techniques could be used to
directly learn the prognostically relevant features in microscopy images of the tumour, without prior identica-
tion of the known tissue entities, e.g. mitoses, nuclear pleomorphism, inltrating immune cells, tumour budding.
Our hypothesis is that training a machine learning classier by using patient outcome as the endpoint could
reveal known prognostic morphologies, but also has the potential to identify previously unknown prognostic
Tissue images typically comprise a combination of a complex set of patterns and conventional design
of an automated tissue classifier requires substantial domain expertise to plan which particular features
to extract and feed into a classification algorithm. This task, known as feature engineering, is often labori-
ous and time-consuming. Deep learning eliminates feature engineering and can learn representative features
Institute for Molecular Medicine Finland FIMM, Helsinki Institute for Life Science HiLIFE, University of Helsinki,
Department of Women’s and Children’s Health, International Maternal and Child Health (IMCH),
Uppsala University, Uppsala, Sweden.
Department of Pathology, Medicum, University of Helsinki, Helsinki, Finland.
Department of Pathology, University of Helsinki and HUSLAB, Helsinki University Hospital, Helsinki, Finland.
Nueld Department of Surgical Sciences, NIHR Oxford Biomedical Research Centre, University of Oxford, Oxford,
Department of Surgery, University of Helsinki and Helsinki University Hospital, Helsinki, Finland.
Programs Unit, Translational Cancer Biology, University of Helsinki, Helsinki, Finland.
Department of Public Health
Sciences, Global Health/IHCAR, Karolinska Institutet, Stockholm, Sweden. Nina Linder and Riku Turkki contributed
equally to this work. Correspondence and requests for materials should be addressed to D.B. (email: dmitrii.
Received: 16 August 2017
Accepted: 12 February 2018
Published: xx xx xxxx