Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You and Your Team.

Learn More →

A Review of Publicly Available Automatic Brain Segmentation Methodologies, Machine Learning Models, Recent Advancements, and Their Comparison

A Review of Publicly Available Automatic Brain Segmentation Methodologies, Machine Learning... Background: The noninvasive study of the structure and functions of the brain using neuroimaging techniques is increasingly being used for its clinical and research perspective. The morphological and volumetric changes in several regions and structures of brains are associated with the prognosis of neurological disorders such as Alzheimer’s disease, epilepsy, schizophrenia, etc. and the early identification of such changes can have huge clinical significance. The accurate segmentation of three-dimensional brain magnetic resonance images into tissue types (i.e., grey matter, white matter, cerebrospinal fluid) and brain structures, thus, has huge importance as they can act as early biomarkers. The manual segmentation though considered the “gold standard” is time-consuming, subjective, and not suitable for bigger neuroimaging studies. Several automatic segmentation tools and algorithms have been developed over the years; the machine learning models particularly those using deep convolutional neural network (CNN) architecture are increasingly being applied to improve the accuracy of automatic methods. Purpose: The purpose of the study is to understand the current and emerging state of automatic segmentation tools, their comparison, machine learning models, their reliability, and shortcomings with an intent to focus on the development of improved methods and algorithms. Methods: The study focuses on the review of publicly available neuroimaging tools, their comparison, and emerging machine learning models particularly those based on CNN architecture developed and published during the last five years. Conclusion: Several software tools developed by various research groups and made publicly available for automatic segmentation of the brain show variability in their results in several comparison studies and have not attained the level of reliability required for clinical studies. The machine learning models particularly three dimensional fully convolutional network models can provide a robust and efficient alternative with relation to publicly available tools but perform poorly on unseen datasets. The challenges related to training, computation cost, reproducibility, and validation across distinct scanning modalities for machine learning models need to be addressed. Keywords Neuroimaging, automatic brain segmentation, FreeSurfer, FSL, SPM, CNN, machine learning (GM), white matter (WM), hippocampus, thalamus, Introduction amygdala, etc., has found greater interest in neuroscience studies for research and clinical intervention. Medical image The advancements in the field of neuroimaging particularly segmentation is used for addressing a wide range of such structural magnetic resonance imaging (MRI) had resulted in the availability of high-resolution three-dimensional (3D) imaging data of the human brain. Several large-population 1 National Brain Research Centre, Manesar, Gurugram, Haryana, India studies using MRI datasets have found different morphometric Symbiosis Centre for Information Technology, Hinjawadi, Pune, Maharashtra, India and volumetric changes in normal vs. various neurological conditions such as Alzheimer’s disease, mild cognitive Corresponding author: 1,2 impairment, etc. Mahender Kumar Singh, National Brain Research Centre, NH-8, Manesar, The study of different areas of the brain, cortical and Gurugram, Haryana 122052, India. subcortical regions and structures, changes in grey matter E-mail: mks.nbrc@gov.in Creative Commons Non Commercial CC BY-NC: This article is distributed under the terms of the Creative Commons Attribution- NonCommercial 4.0 License (http://www.creativecommons.org/licenses/by-nc/4.0/) which permits non-Commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https:// us.sagepub.com/en-us/nam/open-access-at-sage). 2 Annals of Neurosciences biomedical research problems. The accurate segmentation of comparison studies for segmentation accuracy between MRI images thus becomes a necessary prerequisite for the different methods during the last five years. The recent quantitative analysis of the study of neurological diseases machine learning models, particularly those based upon deep including their diagnosis, progression, and treatment learning/CNN architectures that have been proposed by monitoring in a wide variety of neurological disorders such as various research groups for efficient and accurate Alzheimer’s, dementia, focal epilepsy, multiple sclerosis, etc. segmentation, have also been covered. The discussion covers Apart from whole-brain segmentation into GM, WM, the advantages and disadvantages of traditional software cerebrospinal fluid (CSF), etc. and parcellation of brain tools and deep learning machine models and emphasizes structures, several regions, and structures have received future challenges in the area. added focus on the account of their association with important The preprocessing steps which are commonly applied cognitive functions. The hippocampus which is a part of the before any segmentation algorithm are detailed as follows. limbic system that is involved in learning and memory is one such important structure. The MRI can be used to monitor Data Samples morphological changes that occur in the hippocampus in diseases such as Alzheimer’s, schizophrenia, epilepsy, The segmentation is performed on MRI scans of individual depression, etc. and can act as biomarkers for several brain subjects which are acquired on an MRI scanner, and a single disorders such as Alzheimer’s disease, schizophrenia, structural acquisition can take up to 10 to 30 min based upon 5-8 epilepsy, etc. The amygdala is another brain structure of the the precision, sequence, nature of studies, scanner limbic system that is involved in emotion, learning, memory, characteristics, etc. Most studies for the evaluation of the attention, etc. It is also associated with negative emotions and automatic segmentation methods, however, use datasets from fear. The caudate nucleus and putamen also have importance existing available institutional neuroimaging datasets or in several neurological disorders and diseases such as publicly available neuroimaging repositories. Several huge Parkinson’s disease, Huntington’s disease, Alzheimer’s neuroimaging datasets created as a part of the Human 10-13 disease, depression, schizophrenia, etc. Connectome Project, Alzheimer’s Disease Neuroimaging The manual segmentation is considered the “gold standard” Initiative (ADNI), Open Access Series of Imaging Studies for anatomical segmentation. It involves the demarcation of (OASIS), etc. have been extensively used for standardizing structure, grey and white matter along the anatomical boundaries segmentation techniques and finding the correlation between for each layer of the region of interest (ROI) by experienced morphometric/anatomical and volumetric changes in different anatomic tracers utilizing specific software tools. A single regions of the brain. The T1-weighted (T1w) MRI sequence MRI scan can have up to hundreds of layers depending upon the is used for anatomical characterization; however, other scan resolution, and thus manual segmentation is a time- sequences like T2w, FLAIR, etc. are also used by certain consuming, subjective, laborious process and is not suitable for algorithms. The methods which are discussed and evaluated 15,16 large-scale neuroimaging studies. in this review have greatly benefitted from the availability of The automatic segmentation techniques attempt to address such datasets. the above-discussed limitations associated with manual segmentation. The advancement in the algorithms and Preprocessing computation resources over the years has further helped the development of various segmentation techniques. Some of the The acquired MRI images coming from a magnetic resonance hugely used publicly available software tools for neuroimaging (MR) scanner require a series of steps to improve the quality studies that are used for the automatic segmentation of brain of brain scans for the application of segmentation algorithms. regions and structure are (a) FreeSurfer, (b) FMRIB Software These preprocessing steps are also implemented in several Library (FSL), and (c) Statistical Parametric Mapping (SPM). publicly available neuroimaging tools which are often utilized The machine learning approaches based on CNN architecture for carrying out the preprocessing. Some of the preprocessing are also being increasingly used for automated segmentation. steps are discussed as follows. The remaining portion of the review will discuss the models for automatic segmentation techniques, publicly Bias-Field Correction available neuroimaging tools, and emerging machine learning The bias-field or illumination artifact arises on account of a models for automatic segmentation and their comparison. lack of radio-frequency (RF) homogeneity. Although this is not significantly noticeable on visual examination, but it can Methods seriously degrade the volumetric quantification of MR volume upon applying the automatic segmentation algorithms that use The study outlines the literature review of different publicly intensity levels. Several methods for bias-field correction available segmentation methodologies that are extensively during and after acquisition are in use. The phantom-based used for automatic human brain segmentation along with calibration, multi-coil imaging, and special sequences tend to Singh and Singh 3 improve the acquisition process and can be referred to as Automatic Segmentation Approaches prospective methods, whereas retrospective methods and algorithms such as filtering, masking, intensity, gradient, etc. The MRI scan of the brain provides a 3D image of the brain are used postacquisition for bias-field correction. scanned in x, y, z space at an appropriate slice of thickness usually ranging from 1 to 2 mm (e.g., a slice thickness of 1 mm × 1 mm × 1 mm is considered quite good). The slice Brain Extraction thickness need not be isometric and will depend upon the MR The brain extraction or “removal of nonbrain tissue” such as scanner, gradient coil, channels, scan time, scanning sequence, skull, neck, muscles, bones, fat, etc. having overlapping and protocol. Every point in this 3D image “I” represents a intensities is employed before the segmentation processing. voxel “I (x, y, z).” A higher spatial resolution will bring The extraction process classifies the image voxels into the greater precision but require a longer scan time and may not brain or nonbrain; the brain regions can then be extracted or a be convenient for the patient/subject apart from the binary mask can be created for the brain region. The possibilities of the introduction of further noise from head commonly used methods make use of probabilistic atlas movements during the longer scan session. The 3D brain creating a deformable template registered with the image for structure can also be represented as a sequence of 2D images the removal of nonbrain tissue using the brain mask . The in (x, y), (y, z), or (x, z) planes. Every voxel has an intensity brain extraction tool which is a part of the FMRIB Software value typically represented in grayscale from 0 to 255. The Library and which uses the center of gravity of the brain, images may also require to be resampled to a standard space inflating a sphere until the brain boundary is found, provides for applying segmentation algorithms. 21-23 a fast and efficient alternative . Pincram uses an atlas- based label propagation method for brain extraction. Segmentation Algorithms Several software tools, however, do not require separate preprocessing, bias correction, brain extraction, etc. as they The segmentation problem requires classifying every voxel are impliedly included in their segmentation processing. into a specific tissue class such as GM, WM, and CSF, and Figure 1. A Schematic Representation of Different Stages of Brain Segmentation as Obtained from FreeSurfer Tutorial Data and Visualized in FreeSurfer-Freeview. 4 Annals of Neurosciences also to identify and describe it into a specific anatomical Major Software Tools and Their structure by assigning a label corresponding to each voxel. Methodology Several computational algorithms for automatic segmentation have been proposed and used in isolation or conjunction with Several software tools have been developed over the years by each other. The basic methods such as thresholding, fuzzy different scientific groups applying various algorithms for c-means, etc. which are used to segment the brain among automatic segmentation. A list of several publicly available different tissue classes (GM, WM, CSF) are not suitable for neuroimaging software and tools is available in Table 1 with fine-grain segmentation. The deformation models, multi-atlas their brief descriptions. The list is not exhaustive. The segmentation, model-based segmentation, and machine learning important ones are discussed as follows. models, particularly those based upon CNN, are increasingly being used for fine-grain segmentation of the whole brain or ROIs such as the hippocampus, cerebellum, amygdala, etc. FreeSurfer The theoretical concepts behind various segmentation 19,25,26 27 methodologies have already been detailed in the literature. FreeSurfer is an open-source software suite developed by the The present work focuses on developed models, machine Laboratory of Computational Neuroimaging at the Athinoula learning methods, and tools which have been applied to A. Martinos Center for Biomedical Imaging. The software address the automatic segmentation problem. package is freely available from its website https://surfer.nmr. Table 1. Important Publicly Available Software and Tools for Neuroimaging Studies Software Tools Features FreeSurfer • Free and open source. • Linux and Mac platform. • Analysis and visualization of structural and functional neuroimaging data. • Segmentation uses image intensity and probabilistic atlas with local spatial relationships (atlas- based segmentation). • The current version of FreeSurfer is version 7.0 (May 2020). Statistical Parametric Mapping • Free and open source but requires MATLAB or to use the compiled version. (SPM) • Linux/Unix, Mac, and Windows platform. • The current version of SPM is SPM-12 released in January 2020. • Uses tissue probability maps, segmentation, and labeling functionality further enhanced with toolboxes such as VBM8, CAT12, and AAL3. FMRIB Software Library (FSL) • Free and open source. • Linux and Mac platform. • Analysis tools for FMRI, MRI, and DTI neuroimaging data. • Segmentation is done using FSL-FAST for tissue segmentation and FSL-FIRST for subcortical segmentation (model-based). • The current version of FSL is version 6.0. volBrain • Online MRI brain volumetry system. • Provides volumes of GM, WM, CSF as well as macroscopic areas, also provides subcortical structure segmentation and related volumes. • volBrain web platform also provides additional pipelines like • CERES (CEREbellum Segmentation) uses multi-atlas nonlocal patch-based label fusion • HIPS (HIPpocampus subfield Segmentation) • pBrain (Parkinson related deep nucleus segmentation) Multi-atlas propagation with • Code is available through GitHub. enhanced registration (MAPER) • Uses databases of multiple atlases as the knowledge base. • Requires pincram and FSL for brain extraction and tissue class segmentation. Multi-atlas-based multi-image • Uses multi-atlas-based segmentation algorithm. segmentation (MABMIS) • Uses tree-based group-wise registration of atlas and target images. • Performs the simultaneous segmentation of all available images. • Available at https://www.nitrc.org/projects/mabmis • Last version was released in 2011. Automatic segmentation of hip- • Segmentation of the hippocampus subfield using the included atlas. pocampus subfield (ASHS) • It also allows building own atlas and training it to be used for segmentation. • Can be re-trained and extended to other segmentation. • Free and open source, available for Linux and Mac OS. • Available at https://www.nitrc.org/projects/ashs and last version was released in 2017. Singh and Singh 5 mgh.harvard.edu and is used for a complete range of analysis exists. The tissue classification methodology in SPM considers the of structural and functional imaging data and its visualization. brightness information of voxels along with tissue probability The latest version of the software is version 7.1.0 (released in maps and the position of voxel during classification. The latest May 2020). The segmentation pipeline of FreeSurfer can be version is SPM12 and is last updated in January 2020. The run in a fully automatic manner using the “recon-all all” script. segmentation process of SPM can be further extended using SPM It uses image intensity and probabilistic atlas with local spatial toolboxes; the VBM is a toolbox which has been used in several relationships between subcortical structures for carrying out studies and has now been replaced with the CAT12 toolbox. In a the segmentation. The default pipeline can segment and recent study, it was noticed that CAT12 can contribute better in assign 40 different labels to corresponding voxels in an volumetric analysis than VBM8; this is also on account of automated manner. The segmentation has been extended to normalization and segmentation improvements in SPM12. The further segment 9 amygdala nuclei and 13 hippocampus automated anatomical labeling atlas 3 (AAL3) is another toolbox subfields that have been incorporated in the new statistical of SPM which is a refinement of its previous versions and can atlas based on Bayesian inference build using a postmortem parcellate the brain among 166 labels. specimen at high resolution and has been added in FreeSurfer version 6.0 onward. This allows the automatic simultaneous volBrain (Online Web Platform) segmentation of amygdala nuclei and hippocampus subfields using standard resolution structural MR images. The software volBrain is a web-based pipeline for MRI brain volumetry. is freely available for Linux and Mac platforms, and it provides The pipeline can be accessed from the https://volbrain.upv.es/. graphic as well as command line options. The webserver platform allows researchers to submit their MRI scans in NIFTI format and the online pipeline generates the volumetric measurements which are sent on the registered FMRIB Software Library (FSL) email id after the completion of the job; the same can also be 23,31 FSL is a comprehensive library of neuroimaging tools for downloaded from the volBrain system. The user is required to structural, functional, and diffusion tensor imaging (DTI) register with the volBrain online system and there is a limit on studies. The software has been created and maintained by the number of simultaneous jobs that can be submitted. The FMRIB Analysis Group at the University of Oxford and is entire processing remains a black box from the user’s point of available at https://fsl.fmrib.ox.ac.uk/fsl; the current version view; the theoretical aspects have been described in published of the software is version 6.0. The tissue segmentation is done papers. The volBrain system is primarily based on a multi- using FMRIB Automated Segmentation Tool (FAST), atlas, patch-based segmentation method and utilizes training which uses Markov random field model along with the libraries as implemented in the volBrain online platform. The expectation-maximization algorithm. FAST can be invoked volBrain web platform has also added several brain structure through the command line or GUI in the brain-extracted segmentation methods such as CERES (CEREbellum image volumes. The segmentation pipeline of FSL can also 38 39 Segmentation), HIPS (HIPpocampus Segmentation), and correct RF inhomogeneity and classify the brain among GM, pBrain (Parkinson related deep nucleus segmentation), WM, and CSF tissue types. The probabilistic and partial which are all available from the volBrain platform. volume tissue segmentation used for tissue volume calculation can also be calculated. The subcortical segmentation is Multi-Atlas Propagation With Enhanced performed using FMRIB Integrated Registration and 33 Registration (MAPER) Segmentation Tool (FIRST). FIRST provides model-based segmentation using deformation models. The construction of The multi-atlas propagation with enhanced registration the model was based upon a manually labeled dataset; the (MAPER) is an automatic segmentation tool for structural labels were parameterized as surface meshes modeled as a MRI images into corresponding anatomical sub regions by point distribution model. It first registers the images to 41 using a database of multiple atlases as knowledge base. The MNI152 space by performing affine registration. The “run_ standard MAPER pipeline requires brain extraction and tissue first_all” script does the automatic segmentation of subcortical class segmentation using pincram and FSL-FAST, respectively. structures into 15 different labels. Comparison of Segmentation Accuracy in Statistical Parametric Mapping (SPM) Automatic Methods SPM is a package developed for the analysis of neuroimaging Several studies have been made to judge the relative data coming from several imaging modalities like functional- comparison of various publicly available software tools and MRI, positron emission tomography, magnetoencephalography, algorithms. The FreeSurfer, SPM, and FSL are among the most electroencephalography, etc. This is also a freely available used tools for neuroimaging studies and are not limited to software tool from its website https://www.fil.ion.ucl.ac.uk/spm/ segmentation alone. The other software tools are built for at Wellcome Centre for Human Neuroimaging but requires specific applications. The comparison studies are generally MATLAB platform, even though a compiled version of SPM also 6 Annals of Neurosciences performed on specific datasets with default options and provide Table 2 summarizes the recent comparison of several publicly mixed results which cannot be generalized for all situations. available tools for their accuracy in automatic segmentation. Table 2. Comparison of Some Publicly Available Methods for Automatic Segmentation Research Automatic Seg- Citation mentation Tools Dataset Utilized Conclusion/Findings Yaakub et • MAPER • Hammers_mith brain • Methods applied to the three atlas databases of T1-weighted images. al. • FreeSurfer atlas • Leave-one-out-cross-comparison was done for estimating the seg- (5.3) • Mindboggle-101 da- mentation accuracy of methods. tabase (DKT40 atlas • Both identified known abnormalities in the patient groups. database) • FreeSurfer performed superiorly in AD and Left-HS, whereas MAPER • Atlas created from in the Right-HS dataset. OASIS database for • MAPER performed better in healthy controls. MICCAI 2012 grand challenge Palumbo • SPM-12 • Kirby-21 • GM, WM, subcortical structure segmentation in test-retest MRI data et al. • FreeSurfer • OASIS datasets of healthy volunteers. (6.0) • SPM was found more consistent in the evaluation of ROI volume for intra-method repeatability and inter method reproducibility. Bartel et • FASTSURF • Multicentre phase-III • FASTSURF is a semi-automatic contouring-based segmentation al. • FSL-FIRST trial dataset of SCLC model for the hippocampus and uses a mesh processing technique. • FreeSurfer patients • Comparison of hippocampal atrophy rates was made with manual, • ADNI database FreeSurfer, and FSL. • Semi-automatic FASTSURF model was found superior to compared automatic models. Velasco- • FreeSurfer OASIS dataset • The comparison was made in terms of reproducibility and accuracy Annis et • FSL-First n = 20 (scanned twice), for hippocampus, putamen, thalamus, caudate, pallidum, amygdala, ac- al. • PSTAPLE 1.5T cumbens, and brainstem. (Local MAP • PSTAPLE was found to have superior reproducibility. PSTAPLE) Zandifar • FreeSurfer ADNI database • Applied on hippocampus volumes. et al. 5.3 • All methods show acceptable conformity with manual segmentation. • ANIMAL • Patch-based strategies have a good correlation with manual segmen- • Patch-based tation. methods Perlaki et • FSL-FIRST 30 healthy young Caucasian • Study to compare the segmentation accuracy of the caudate nucleus al. • FreeSurfer subjects and putamen. (v4, 5, and n = 30, 3T • FSL was found to be superior for putamen segmentation. 5.3) Naess- • FreeSurfer • 22 healthy subjects • Thalamus and hippocampus automatic segmentation. Schmidt (5.3) (age 19–40) n = • volBrain (patch-based) provided more accuracy in MP2RAGE images et al. • FSL-FIRST 22, 3T than conventional ones. (4.1.9) • MP2RAGE, for DTI • SPM-12 n = 10 • volBrain Grimm et • FreeSurfer 92 participants in the • Automatic methods are compared with manual segmentation for al. • VBM age range of (18–34, amygdalar and hippocampus volume. mean:21.64) • Both methods were found comparable to manual segmentation. n = 92, 1.5T Fellhauer • FreeSurfer n = 115, 1.5T, MPRAGE • All three detected increase in brain atrophy in AD/MCI group. et al. (5.1) (60 MCI, 34 AD, 32 healthy) • FSL was good with good quality images. • FSL-FAST • SPM was recommended for patient data in difficult measurement (4.1.9) situations. • SPM 8 and 12 (with VBM8) Abbreviations: MCI, mild cognitive impairment Singh and Singh 7 The recent methods of automatic machine learning models Evolving Machine Learning Models for of segmentation are discussed as follows. Automatic Segmentation Several machine learning models have been developed over AssemblyNet the years for the automatic segmentation of brain tissue and This model uses a large assembly of CNNs, each processing anatomical structures of specific structures like the cerebellum different overlapping regions. The framework is arranged in and hippocampus. Fine-grain segmentation of the whole the form of two assemblies of U-Nets, each having 125 3D brain has also been implemented in several methods. A U-Nets (i.e., a total of 250 compact 3D U-Nets). The method summary of some recent machine learning models applied for showed competitive performance in respect of U-Net, patch- automatic brain segmentation is given in Table 3. based joint label fusion, and SLANT-27 methods. The The regular machine learning models do not generalize segmentation accuracy was improved with the use of well and are not suitable for complex imaging modalities; the nonlinearly registered “Atlas prior” for expected classification, deep learning models having multiple layers are increasingly transfer learning to initialize spatially nearest U-Net and being used to address neuroimaging challenges which have multi-scale cascade to communicate between the two benefitted from the advancements in graphical processing assemblies for refinement. The training time was shown to be unit (GPU) processing power. The deep learning models like 7 days in the case of AssemblyNet but the classification time CNN are preferred owing to their application in medical is just 10 min. imaging problems. A typical CNN architecture contains several layers and components such as (a) input layers receiving raw image data, (b) convolutional layers applying SLANT filter (kernel) and producing feature maps, (c) activation The SLANT pipeline first registers the target image to the function layer applying the activation function to the output MNI305 template using affine registration, and this is coming from the convolution layer like rectified linear unit, followed by bias-field correction and normalization. Since (d) pooling layers for downsampling the output of the the whole volume cannot fit into the GPU memory using the preceding layer, and (e) fully connected layer for applying FCN network, the entire MNI space is divided into k weights to feature analysis to predict the label. The typical independent 3D U-Nets. The SLANT-8 divides the volume CNN models are extended in various CNN architectures such 53 54 into eight nonoverlapped subspaces with each subspace of as U-Nets. 3D U-Nets is an extension of traditional U-Net size 86 × 110 × 78 voxels, whereas SLANT-27 divides the architecture for application in 3D biomedical imaging. Figure 2. An Illustration of AssemblyNet Model Using U-Net . Source: Reprinted with permission from Elsevier. 8 Annals of Neurosciences Figure 3. An Illustration of SLANT-27 (27-Network Tiles) Model Using U-Net . (Reprinted with permission from Elsevier). Source: Reprinted with permission from Elsevier. volume into 27 overlapped network tiles with each subspace capacity of the model. The model was successfully applied of 96 × 128 × 88 voxels. The overlapped SLANT-27 thus for 3D whole-brain segmentation and achieved comparable requires majority voting for label fusion. The entire pipeline performance with up to 16 times model compression. is available as a docker image. It provides fine-grain segmentation of the whole brain in more than 100 ROIs. The QuickNAT training time could be as high as 109 h for SLANT-27 on 5,111 training scans using a single GPU (NVIDIA Titan 12 QuickNAT tries to address the limitation of limited manually GB), which can be reduced to 4 h with 27 GPUs, and the curated data for training by first utilizing the existing publicly testing time is roughly 15 min. available software tools such as FreeSurfer to segment the data which is then used to train the network; this pretrained network is then further improved by training with limited 3DQ manually annotated data to achieve high segmentation This is a generalizable method that can be applied to various accuracy. The model does not utilize 3D F-CNN but instead 3D F-CNN architecture such as 3D U-Nets, MALC, and uses three two-dimensional F-CNN, each operating in “V-Net on MALC” for providing model compression of 16 separate planes (i.e., coronal, axial, and sagittal), followed by times to address the memory issues, and it incorporates aggregation resulting in final segmentation and labels quantization mechanism for integrating the trainable scaling corresponding to 27 brain structures. factor and the normalization parameter, which not only The details of other recent methods and architecture have maintains compression but also increases the learning been given in Table 3. Singh and Singh 9 Table 3. Summary of Recent Brain Segmentation Methods Utilizing Machine Learning Models Model/Segmentation Method Area Principle Remarks A. Whole-Brain Segmentation Models for Classifying Among Different Anatomical Labels AssemblyNet CNN/ Utilizes two assemblies of 125 Competitive performance in compari- Brain automatic segmentation 3D U-Nets processing different son with U-Net, joint label fusion, and overlapping brain areas of the SLANT. whole brain. SLANT CNN/ 3D–FCN, addresses memory Training and testing can be optimized by Fine-grain segmentation >100 issues using multiple spatially dis- providing 27 GPUs for SLANT-27 and 8 structures tributed overlapping network tiles for SLANT-8. of U-Nets. QuickNAT CNN/ Fully convolutional and densely Posttraining, the model achieves su- Brain segmentation connected, perior computational performance in (segments 27 structures) pretraining using existing segmen- comparison with patch-based CNN and tation software (FreeSurfer), atlas-based approaches. Also compared fine-tuning to rectify errors using well with FSL and FreeSurfer. manual labels. Bayesian QuickNAT CNN/ F-CNN approach (of QuickNAT) The model has been compared with Brain segmentation with Bayesian inference for seg- QuickNAT and FreeSurfer with manual (segments 33 structures) mentation quality. annotations. 3DQ CNN/ 3D F-CNN with model compres- Integrates training scalable factors and Brain segmentation sion up to 16 times without affect- normalization parameter. (segment 28 structures) ing performance. Useful for storage Increases learning while maintaining critical applications. compression. DeepNAT CNN/ 3D-CNN patch-based model, Uses three CNN layers for pooling, Brain segmentation of 25 the first network removes the normalization, and nonlinearities. structures background, second classifies brain Comparable with other state-of-the-art structure. models. BrainSegNet CNN/ 2D/3D CNN patches Does not require registration, saving on Whole-brain segmentation computational cost. B. Whole-Brain Segmentation Models for Classification Among GM, WM, and CSF Only HyperDense-Net CNN / Fully connected 3D-CNN using Successfully participated in iSEG-2017 Brain segmentation multiple modalities. and MRbrainS-2013 challenge. VoxResNet CNN/ Voxel-wise residual network with Successfully competed in Brain segmentation 25 layers utilizing CNN. MRbrainS-2015 challenge. C. Brain Segmentation Models for Specific Brain Structures (Hippocampus, Cerebellum) HippMapp3r CNN/ CNN architecture based upon Validation is done against FreeSurfer, Hippocampus segmentation U-Net. FSL-First, volBrain, SBHV, and Hip- Initial training on the whole brain, poDeep. Algorithm and trained model the output was trained again with are made publicly available. reduced FOV on the same net- work architecture. CAST CNN/Hippocampus subfield Multi-scale 3D CNN with T1w and Hippocampus subfield segmentation. segmentation T2w imaging modalities as input. ACA-PULCO CNN/ Alternative CNN design using Applied on MPRAGE images. Cerebellum segmentation U-Net with locally constrained optimization. HippoDeep CNN/ Deep learned appearance model The training utilizes multiple cohorts Hippocampus automatic based on CNN. and label derived from FreeSurfer out- segmentation put along with synthetic data. 10 Annals of Neurosciences imaging volumes. Several techniques have been developed to Reliability and Reusability of Automated address the computational cost and memory limitations; for Segmentation Methods example, SLANT-27 breaks the original image volume into 27 overlapping network tiles which can be executed in The automated fine-grain segmentation of the whole brain parallel on 27 GPUs or can run sequentially in the available including specific ROIs has profound usage in neuroscience lesser resources; AssemblyNet goes a step further and uses research. This, however, depends on the quality of MRI two assemblies of 125 3D U-Nets processing different acquisitions, preprocessing, choice of analysis pipelines, and overlapping regions; 3DQ method attempts to provide around many other factors. A retrospective longitudinal study has 16 times compression without affecting performance; other shown a lack of reproducibility arising at the level of methods such as ACA-PULCO and CAST focus on specific acquisition in the MRI scanner and the amount of variability 68 regions such as cerebellum and hippocampus, respectively, being different for different scanners, and this is a cause of rather than whole-brain segmentation. concern having a direct bearing on subsequent analysis Other issues with the machine learning models are with pipelines. The effects of automated pipelines which are often regard to insufficient documentation in the literature; only poorly documented and lack standardization have also been some such methods are publicly made available, even then analyzed in a recent study in the context of functional the training modalities of some of them especially those neuroimaging analysis where the same data were provided to relying on 3D-FCN have a huge computational and memory 70 different research groups, but none of the teams chose bottleneck associated with them and cannot be easily identical pipelines, and the results were also variable across reproduced. There is a need to implement such machine groups, thus emphasizing the need of validation and sharing 69 learning models through computational webservers in a of complex workflows. standardized manner as was the case with volBrain for their effective use and validation. Discussion and Conclusion Acknowledgments The automatic segmentation methods have been quite The authors like to thank Professor Pankaj Seth, National Brain successful in morphometric and volumetric measurements of Research Centre for his guidance, support, and help, including brain tissue and structures in finer details. The publicly proofreading of the manuscript. The support of Dr D. D. Lal, Mr available methods such as FreeSurfer, FSL, SPM, etc. are Sukhkir, and DBT e-Library Consortium (DeLCON) in making sufficiently resilient with respect to noise and artifacts available the research articles is also duly acknowledged. The work introduced at the acquisition stage and have performed has also benefitted from the infrastructure facilities under DIC project consistently across different datasets and are being extensively of BTISNet programme and Dementia Science programme, both used by the neuroimaging community; however, they are still funded by the Department of Biotechnology, Government of India. far from being accepted at par with manual segmentation. Several studies have compared the relative performance of Author Contributions various publicly available methods which have provided MKS conceived the study and was responsible for overall study mixed results and have been discussed in Table 2. direction and planning; KKS provided direction, suggestion, and The machine learning methods particularly those based on supervision. Both the authors discussed the final manuscript. CNN have been shown to perform better than the publicly available software tools, but their performance directly Ethical Statement depends on the amount and type of training data and thus may Not Applicable. not be reproducible across different unseen datasets whose acquisition and protocols vary significantly from the training Declaration of Conflicting Interests data. The insufficient amount of manually labeled training data also affects the performance of such machine learning The authors declared no potential conflicts of interest with respect to models. QuickNAT attempts to improve upon this limitation the research, authorship, and/or publication of this article. by first pretraining the network using auxiliary labels generated using FreeSurfer and then refining the pretrained Funding network with the limited manually labeled data. The authors received no financial support for the research, The 2D F-CNN methods which train the network by using authorship, and/or publication of this article. images slice by slice have an inherent disadvantage of failure to fully utilize the contextual information from neighboring ORCID iDs slices. The 3D F-CNN, on the other hand, faces the memory Mahender Kumar Singh https://orcid.org/0000-0001-9617-6112 limitation of the available GPUs in handling millions of parameters associated with high-resolution clinical MRI Krishna Kumar Singh https://orcid.org/0000-0003-3849-5945 Singh and Singh 11 References tions. Comput Math Methods Med; 2015 ; 2015: 450341. DOI: 10.1155/2015/450341. 1. Driscoll I, Davatzikos C, An Y, et al. Longitudinal pattern of 20. Xue H, Srinivasan L, Jiang S, et al. Automatic segmentation and regional brain volume change differentiates normal aging from reconstruction of the cortex from neonatal MRI. NeuroImage MCI. Neurology 2009; 72: 1906–1913. 2007; 38: 461–477. 2. Franke K and Gaser C. Longitudinal changes in individual 21. Smith SM. Fast robust automated brain extraction. Hum Brain BrainAGE in healthy aging, mild cognitive impairment, and Mapp 2002; 17: 143–155. Alzheimer’s disease. GeroPsych 2012; 25: 235–245. 22. Battaglini M, Smith SM, Brogi S, et al. Enhanced brain 3. Raghuprasad MS and Manivannan M. Volumetric and mor- extraction improves the accuracy of brain atrophy estimation. phometric analysis of pineal and pituitary glands of an Indian NeuroImage 2008; 40: 583–589. inedial subject. Ann Neurosci 2018; 25: 279–288. 23. Jenkinson M, Beckmann CF, Behrens TEJ, et al. FSL. 4. Giorgio A and de Stefano N. Clinical use of brain volumetry. J NeuroImage 2012; 62: 782–790. Magn Reson Imaging 2013; 37: 1–14. 24. Heckemann RA, Ledig C, Gray KR, et al. Brain extraction 5. Wu W-C, Huang C-C, Chung H-W, et al. Hippocampal altera- using label propagation and group agreement: Pincram. PLoS tions in children with temporal lobe epilepsy with or without a ONE 2015; 10: 1–18. history of febrile convulsions: Evaluations with MR volumetry 25. Yazdani S, Yusof R, Karimian A, et al. Image segmentation and proton MR spectroscopy. AJNR Am J Neuroradiol 2005; methods and applications in MRI brain images. IETE Tech Rev 26: 1270–1275. 6. Apostolova LG, Dinov ID, Dutton RA, et al. 3D comparison 2015; 32: 413–427. of hippocampal atrophy in amnestic mild cognitive impairment 26. Iglesias JE and Sabuncu MR. Multi-atlas segmentation of bio- and Alzheimer’s disease. Brain 2006; 129: 2867–2873. medical images: A survey. Med Image Anal 2015; 24: 205–219. 7. Tanskanen P, Veijola JM, Piippo UK, et al. Hippocampus and 27. Fischl B. FreeSurfer. NeuroImage 2012; 62: 774–781. amygdala volumes in schizophrenia and other psychoses in the 28. Fischl B, Salat DH, Busa E, et al. Whole brain segmentation: Northern Finland 1966 birth cohort. Schizophr Res 2005; 75: Automated labeling of neuroanatomical structures in the human 283–294. brain. Neuron 2002; 33: 341–355. 8. Bremner JD, Narayan M, Anderson ER, et al. Hippocampal vol- 29. Saygin ZM, Kliemann D, Iglesias JE, et al. High-resolution ume reduction in major depression. Am J Psychiatry 2000; 157: magnetic resonance imaging reveals nuclei of the human amyg- 115–118. dala: Manual segmentation to automatic atlas. NeuroImage 9. Baxter MG and Murray EA. The amygdala and reward. Nat Rev 2017; 155: 370–382. Neurosci 2002; 3: 563–573. 30. Iglesias JE, Augustinack JC, Nguyen K, et al. A computational 10. Sacchet MD, Livermore EE, Iglesias JE, et al. Subcortical vol- atlas of the hippocampal formation using ex vivo, ultra-high umes differentiate major depressive disorder, bipolar disorder, resolution MRI: Application to adaptive segmentation of in and remitted major depressive disorder. J Psychiatr Res 2015; vivo MRI. NeuroImage 2015; 115: 117–137. 68: 91–98. 31. Smith SM, Jenkinson M, Woolrich MW, et al. Advances in 11. Walker FO. Huntington’s disease. The Lancet 2007; 369: functional and structural MR image analysis and implementa- 218–228. tion as FSL. NeuroImage 2004; 23: S208–S219. DOI: 10.1016/j. 12. Hokama H, Shenton ME, Nestor PG, et al. Caudate, putamen, neuroimage.2004.07.051. and globus pallidus volume in schizophrenia: A quantitative 32. Zhang Y, Brady M, and Smith S. Segmentation of brain MR MRI study. Psychiatry Res Neuroimaging 1995; 61: 209–229. images through a hidden Markov random field model and the 13. Sterling NW, Du G, Lewis MM, et al. Striatal shape in expectation-maximization algorithm. IEEE Trans Med Imaging Parkinson’s disease. Neurobiol Aging 2013; 34: 2510–2516. 2001; 20: 45–57. 14. Shen L, Firpi HA, Saykin AJ, et al. Parametric surface model- 33. Patenaude B, Smith SM, Kennedy DN, et al. A Bayesian model ing and registration for comparison of manual and automated of shape and appearance for subcortical brain segmentation. segmentation of the hippocampus. Hippocampus 2009; 19: NeuroImage 2011; 56: 907–922. DOI: 10.1016/j.neuroim- 588–595. age.2011.02.046. 15. Cherbuin N, Anstey KJ, Réglade-Meslin C, et al. In vivo hip- 34. Ashburner J. SPM: A history. NeuroImage 2012; 62: 791–800. pocampal measurement and memory: A comparison of manual 35. Farokhian F, Beheshti I, Sone D, et al. Comparing CAT12 and tracing and automated segmentation in a large community- VBM8 for detecting brain morphological abnormalities in tem- based sample. PLoS ONE 2009; 4: e5265. poral lobe epilepsy. Front Neurol 2017; 8: 1–7. 16. Wenger E, Mårtensson J, Noack H, et al. Comparing man- 36. Rolls ET, Huang CC, Lin CP, et al. Automated anatomical ual and automatic segmentation of hippocampal volumes: Reliability and validity issues in younger and older brains. Hum labelling atlas 3. NeuroImage 2020; 206: 116189. Brain Mapp 2014; 35: 4236–4248. 37. Manjón JV and Coupé P. volBrain: An online MRI brain volu- 17. Gispert JD, Reig S, Pascau J, et al. Method for bias field cor- metry system. Front Neuroinform 2016; 10: 30. DOI: 10.3389/ rection of brain T1-weighted magnetic images minimizing seg- fninf.2016.00030. mentation error. Hum Brain Mapp 2004; 22: 133–144. 38. Romero JE, Coupé P, Giraud R, et al. CERES: A new cerebellum 18. Song S, Zheng Y, and He Y. A review of methods for bias cor- lobule segmentation method. NeuroImage 2017; 147: 916–924. rection in medical images. Biomed Eng Rev 2017; 3: 1–10. 39. Romero JE, Coupé P, and Manjón J v. HIPS: A new hippo- 19. Despotović I, Goossens B, and Philips W. MRI segmenta- campus subfield segmentation method. NeuroImage 2017; 163: tion of the human brain: Challenges, methods, and applica- 286–295. 12 Annals of Neurosciences tation. In: Ourselin S, Joskowicz L, Sabuncu MR, et al., 40. Manjón J v, Bertó A, Romero JE, et al. pBrain: A novel pipeline eds. Medical Image Computing and Computer-Assisted for Parkinson related brain structure segmentation. NeuroImage Intervention – MICCAI 2016. Springer International Clin 2020; 25: 102184. Publishing, 2016: pp. 424–432. 41. Heckemann RA, Keihaninejad S, Aljabar P, et al. Improving 55. Coupé P, Mansencal B, Clément M, et al. AssemblyNet: A intersubject image registration using tissue-class information large ensemble of CNNs for 3D whole brain MRI segmenta- benefits robustness and accuracy of multi-atlas based anatomi- tion. NeuroImage 2020; 219: 117026. cal segmentation. NeuroImage 2010; 51: 221–227. 56. Huo Y, Xu Z, Xiong Y, et al. 3D whole brain segmentation 42. Jia H, Yap P-T, and Shen D. Iterative multi-atlas-based multi- using spatially localized atlas network tiles. NeuroImage 2019; image segmentation with tree-based registration. NeuroImage 194: 105–119. 2012; 59: 422–430. 57. Paschali M, Gasperini S, Roy AG, et al. 3DQ: Compact quan- 43. Yushkevich PA, Pluta JB, Wang H, et al. Automated volumetry tized neural networks for volumetric whole brain segmenta- and regional thickness analysis of hippocampal subfields and tion. In Shen D. et al. (eds.) Medical Image Computing and medial temporal cortical structures in mild cognitive impair- Computer Assisted Intervention - MICCAI 2019. MICCAI ment. Hum Brain Mapp 2015; 36: 258–287. 2019. Lecture Notes in Computer Science, vol 11766. Springer, 44. Yaakub SN, Heckemann RA, Keller SS, et al. On brain atlas Cham, 2019: pp. 438–446. choice and automatic segmentation methods: a comparison 58. Guha Roy A, Conjeti S, Navab N, et al. QuickNAT: A fully of MAPER & FreeSurfer using three atlas databases. Sci Rep convolutional network for quick and accurate segmentation of 2020; 10: 2837. neuroanatomy. NeuroImage 2019; 186: 713–727. 45. Palumbo L, Bosco P, Fantacci ME, et al. Evaluation of the intra- 59. Roy AG, Conjeti S, Navab N, et al. Bayesian QuickNAT: Model and inter-method agreement of brain MRI segmentation soft- uncertainty in deep whole brain segmentation for structure-wise ware packages: A comparison between SPM12 and FreeSurfer quality control. NeuroImage 2019; 195: 11–22. v6.0. Physica Medica 2019; 64: 261–272. 60. Wachinger C, Reuter M, and Klein T. DeepNAT: Deep con- 46. Bartel F, Vrenken H, van Herk M, et al. Fast segmentation volutional neural network for segmenting neuroanatomy. through surface fairing (FastSURF): A novel semi-automatic NeuroImage 2018; 170: 434–445. hippocampus segmentation method. PLoS ONE 2019; 14: 61. Mehta R, Majumdar A, and Sivaswamy J. BrainSegNet: A con- e0210641. volutional neural network architecture for automated segmenta- 47. Velasco-Annis C, Akhondi-Asl A, Stamm A, et al. tion of human brain structures. J Med Imaging 2017; 4: 24003. Reproducibility of brain MRI segmentation algorithms: 62. Dolz J, Gopinath K, Yuan J, et al. HyperDense-Net: A hyper- Empirical comparison of local MAP PSTAPLE, FreeSurfer, densely connected CNN for multi-modal image segmentation. and FSL-FIRST. J Neuroimaging 2018; 28: 162–172. IEEE Trans Med Imaging 2019; 38: 1116–1126. 48. Zandifar A, Fonov V, Coupé P, et al. A comparison of accurate 63. Chen H, Dou Q, Yu L, et al. VoxResNet: Deep voxelwise automatic hippocampal segmentation methods. NeuroImage residual networks for brain segmentation from 3D MR images. 2017; 155: 383–393. NeuroImage 2018; 170: 446–455. 49. Perlaki G, Horvath R, Nagy SA, et al. Comparison of accuracy 64. Goubran M, Ntiri EE, Akhavein H, et al. Hippocampal segmen- between FSL’s FIRST and Freesurfer for caudate nucleus and tation for brains with extensive atrophy using three-dimensional putamen segmentation. Sci Rep 2017; 7: 2418. convolutional neural networks. Hum Brain Mapp 2020; 41: 50. Næss-Schmidt E, Tietze A, Blicher JU, et al. Automatic 291–308. thalamus and hippocampus segmentation from MP2RAGE: 65. Yang Z, Zhuang X, Mishra V, et al. CAST: A multi-scale con- Comparison of publicly available methods and implications for volutional neural network based automated hippocampal sub- DTI quantification. Int Comput Assist Radiol Surg 2016; 11: field segmentation toolbox: CAST: Segmenting hippocampal 1979–1991. subfields. NeuroImage 2020; 218: 116947. 51. Grimm O, Pohlack S, Cacciaglia R, et al. Amygdalar and hippo- 66. Han S, Carass A, He Y, et al. Automatic cerebellum anatomical campal volume: A comparison between manual segmentation, parcellation using U-Net with locally constrained optimization. Freesurfer and VBM. J Neurosci Methods 2015; 253: 254–261. NeuroImage 2020; 218: 116819. 52. Fellhauer I, Zöllner FG, Schröder J, et al. Comparison of auto- 67. Thyreau B, Sato K, Fukuda H, et al. Segmentation of the hippo- mated brain segmentation using a brain phantom and patients campus by transferring algorithmic knowledge for large cohort with early Alzheimer’s dementia or mild cognitive impairment. processing. Med Image Anal 2018; 43: 214–228. Psychiatry Res Neuroimaging 2015; 233: 299–305. 68. Yang CY, Liu HM, Chen SK, et al. Reproducibility of brain 53. Ronneberger O, Fischer P, and Brox T. U-Net: Convolutional morphometry from short-term repeat clinical MRI examina- networks for biomedical image segmentation. In: Navab N, tions: A retrospective study. PLoS ONE 2016; 11: e0146913. Hornegger J, Wells WM, et al., eds. Medical Image Computing 69. Botvinik-Nezer R, Holzmeister F, Camerer CF, et al. Variability and Computer-Assisted Intervention – MICCAI 2015. Springer in the analysis of a single neuroimaging dataset by many teams. International Publishing, 2015: pp. 234–241. 54. Çiçek Ö, Abdulkadir A, Lienkamp SS, et al. 3D U-Net: Nature 2020; 582: 84–88. Learning dense volumetric segmentation from sparse anno- http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Annals of Neurosciences SAGE

A Review of Publicly Available Automatic Brain Segmentation Methodologies, Machine Learning Models, Recent Advancements, and Their Comparison

Annals of Neurosciences , Volume OnlineFirst: 1 – Jan 1, 2021

Loading next page...
 
/lp/sage/a-review-of-publicly-available-automatic-brain-segmentation-tJUhNkG4HG
Publisher
SAGE
Copyright
© 2021 Indian Academy of Neurosciences (IAN)
ISSN
0972-7531
eISSN
0976-3260
DOI
10.1177/0972753121990175
Publisher site
See Article on Publisher Site

Abstract

Background: The noninvasive study of the structure and functions of the brain using neuroimaging techniques is increasingly being used for its clinical and research perspective. The morphological and volumetric changes in several regions and structures of brains are associated with the prognosis of neurological disorders such as Alzheimer’s disease, epilepsy, schizophrenia, etc. and the early identification of such changes can have huge clinical significance. The accurate segmentation of three-dimensional brain magnetic resonance images into tissue types (i.e., grey matter, white matter, cerebrospinal fluid) and brain structures, thus, has huge importance as they can act as early biomarkers. The manual segmentation though considered the “gold standard” is time-consuming, subjective, and not suitable for bigger neuroimaging studies. Several automatic segmentation tools and algorithms have been developed over the years; the machine learning models particularly those using deep convolutional neural network (CNN) architecture are increasingly being applied to improve the accuracy of automatic methods. Purpose: The purpose of the study is to understand the current and emerging state of automatic segmentation tools, their comparison, machine learning models, their reliability, and shortcomings with an intent to focus on the development of improved methods and algorithms. Methods: The study focuses on the review of publicly available neuroimaging tools, their comparison, and emerging machine learning models particularly those based on CNN architecture developed and published during the last five years. Conclusion: Several software tools developed by various research groups and made publicly available for automatic segmentation of the brain show variability in their results in several comparison studies and have not attained the level of reliability required for clinical studies. The machine learning models particularly three dimensional fully convolutional network models can provide a robust and efficient alternative with relation to publicly available tools but perform poorly on unseen datasets. The challenges related to training, computation cost, reproducibility, and validation across distinct scanning modalities for machine learning models need to be addressed. Keywords Neuroimaging, automatic brain segmentation, FreeSurfer, FSL, SPM, CNN, machine learning (GM), white matter (WM), hippocampus, thalamus, Introduction amygdala, etc., has found greater interest in neuroscience studies for research and clinical intervention. Medical image The advancements in the field of neuroimaging particularly segmentation is used for addressing a wide range of such structural magnetic resonance imaging (MRI) had resulted in the availability of high-resolution three-dimensional (3D) imaging data of the human brain. Several large-population 1 National Brain Research Centre, Manesar, Gurugram, Haryana, India studies using MRI datasets have found different morphometric Symbiosis Centre for Information Technology, Hinjawadi, Pune, Maharashtra, India and volumetric changes in normal vs. various neurological conditions such as Alzheimer’s disease, mild cognitive Corresponding author: 1,2 impairment, etc. Mahender Kumar Singh, National Brain Research Centre, NH-8, Manesar, The study of different areas of the brain, cortical and Gurugram, Haryana 122052, India. subcortical regions and structures, changes in grey matter E-mail: mks.nbrc@gov.in Creative Commons Non Commercial CC BY-NC: This article is distributed under the terms of the Creative Commons Attribution- NonCommercial 4.0 License (http://www.creativecommons.org/licenses/by-nc/4.0/) which permits non-Commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https:// us.sagepub.com/en-us/nam/open-access-at-sage). 2 Annals of Neurosciences biomedical research problems. The accurate segmentation of comparison studies for segmentation accuracy between MRI images thus becomes a necessary prerequisite for the different methods during the last five years. The recent quantitative analysis of the study of neurological diseases machine learning models, particularly those based upon deep including their diagnosis, progression, and treatment learning/CNN architectures that have been proposed by monitoring in a wide variety of neurological disorders such as various research groups for efficient and accurate Alzheimer’s, dementia, focal epilepsy, multiple sclerosis, etc. segmentation, have also been covered. The discussion covers Apart from whole-brain segmentation into GM, WM, the advantages and disadvantages of traditional software cerebrospinal fluid (CSF), etc. and parcellation of brain tools and deep learning machine models and emphasizes structures, several regions, and structures have received future challenges in the area. added focus on the account of their association with important The preprocessing steps which are commonly applied cognitive functions. The hippocampus which is a part of the before any segmentation algorithm are detailed as follows. limbic system that is involved in learning and memory is one such important structure. The MRI can be used to monitor Data Samples morphological changes that occur in the hippocampus in diseases such as Alzheimer’s, schizophrenia, epilepsy, The segmentation is performed on MRI scans of individual depression, etc. and can act as biomarkers for several brain subjects which are acquired on an MRI scanner, and a single disorders such as Alzheimer’s disease, schizophrenia, structural acquisition can take up to 10 to 30 min based upon 5-8 epilepsy, etc. The amygdala is another brain structure of the the precision, sequence, nature of studies, scanner limbic system that is involved in emotion, learning, memory, characteristics, etc. Most studies for the evaluation of the attention, etc. It is also associated with negative emotions and automatic segmentation methods, however, use datasets from fear. The caudate nucleus and putamen also have importance existing available institutional neuroimaging datasets or in several neurological disorders and diseases such as publicly available neuroimaging repositories. Several huge Parkinson’s disease, Huntington’s disease, Alzheimer’s neuroimaging datasets created as a part of the Human 10-13 disease, depression, schizophrenia, etc. Connectome Project, Alzheimer’s Disease Neuroimaging The manual segmentation is considered the “gold standard” Initiative (ADNI), Open Access Series of Imaging Studies for anatomical segmentation. It involves the demarcation of (OASIS), etc. have been extensively used for standardizing structure, grey and white matter along the anatomical boundaries segmentation techniques and finding the correlation between for each layer of the region of interest (ROI) by experienced morphometric/anatomical and volumetric changes in different anatomic tracers utilizing specific software tools. A single regions of the brain. The T1-weighted (T1w) MRI sequence MRI scan can have up to hundreds of layers depending upon the is used for anatomical characterization; however, other scan resolution, and thus manual segmentation is a time- sequences like T2w, FLAIR, etc. are also used by certain consuming, subjective, laborious process and is not suitable for algorithms. The methods which are discussed and evaluated 15,16 large-scale neuroimaging studies. in this review have greatly benefitted from the availability of The automatic segmentation techniques attempt to address such datasets. the above-discussed limitations associated with manual segmentation. The advancement in the algorithms and Preprocessing computation resources over the years has further helped the development of various segmentation techniques. Some of the The acquired MRI images coming from a magnetic resonance hugely used publicly available software tools for neuroimaging (MR) scanner require a series of steps to improve the quality studies that are used for the automatic segmentation of brain of brain scans for the application of segmentation algorithms. regions and structure are (a) FreeSurfer, (b) FMRIB Software These preprocessing steps are also implemented in several Library (FSL), and (c) Statistical Parametric Mapping (SPM). publicly available neuroimaging tools which are often utilized The machine learning approaches based on CNN architecture for carrying out the preprocessing. Some of the preprocessing are also being increasingly used for automated segmentation. steps are discussed as follows. The remaining portion of the review will discuss the models for automatic segmentation techniques, publicly Bias-Field Correction available neuroimaging tools, and emerging machine learning The bias-field or illumination artifact arises on account of a models for automatic segmentation and their comparison. lack of radio-frequency (RF) homogeneity. Although this is not significantly noticeable on visual examination, but it can Methods seriously degrade the volumetric quantification of MR volume upon applying the automatic segmentation algorithms that use The study outlines the literature review of different publicly intensity levels. Several methods for bias-field correction available segmentation methodologies that are extensively during and after acquisition are in use. The phantom-based used for automatic human brain segmentation along with calibration, multi-coil imaging, and special sequences tend to Singh and Singh 3 improve the acquisition process and can be referred to as Automatic Segmentation Approaches prospective methods, whereas retrospective methods and algorithms such as filtering, masking, intensity, gradient, etc. The MRI scan of the brain provides a 3D image of the brain are used postacquisition for bias-field correction. scanned in x, y, z space at an appropriate slice of thickness usually ranging from 1 to 2 mm (e.g., a slice thickness of 1 mm × 1 mm × 1 mm is considered quite good). The slice Brain Extraction thickness need not be isometric and will depend upon the MR The brain extraction or “removal of nonbrain tissue” such as scanner, gradient coil, channels, scan time, scanning sequence, skull, neck, muscles, bones, fat, etc. having overlapping and protocol. Every point in this 3D image “I” represents a intensities is employed before the segmentation processing. voxel “I (x, y, z).” A higher spatial resolution will bring The extraction process classifies the image voxels into the greater precision but require a longer scan time and may not brain or nonbrain; the brain regions can then be extracted or a be convenient for the patient/subject apart from the binary mask can be created for the brain region. The possibilities of the introduction of further noise from head commonly used methods make use of probabilistic atlas movements during the longer scan session. The 3D brain creating a deformable template registered with the image for structure can also be represented as a sequence of 2D images the removal of nonbrain tissue using the brain mask . The in (x, y), (y, z), or (x, z) planes. Every voxel has an intensity brain extraction tool which is a part of the FMRIB Software value typically represented in grayscale from 0 to 255. The Library and which uses the center of gravity of the brain, images may also require to be resampled to a standard space inflating a sphere until the brain boundary is found, provides for applying segmentation algorithms. 21-23 a fast and efficient alternative . Pincram uses an atlas- based label propagation method for brain extraction. Segmentation Algorithms Several software tools, however, do not require separate preprocessing, bias correction, brain extraction, etc. as they The segmentation problem requires classifying every voxel are impliedly included in their segmentation processing. into a specific tissue class such as GM, WM, and CSF, and Figure 1. A Schematic Representation of Different Stages of Brain Segmentation as Obtained from FreeSurfer Tutorial Data and Visualized in FreeSurfer-Freeview. 4 Annals of Neurosciences also to identify and describe it into a specific anatomical Major Software Tools and Their structure by assigning a label corresponding to each voxel. Methodology Several computational algorithms for automatic segmentation have been proposed and used in isolation or conjunction with Several software tools have been developed over the years by each other. The basic methods such as thresholding, fuzzy different scientific groups applying various algorithms for c-means, etc. which are used to segment the brain among automatic segmentation. A list of several publicly available different tissue classes (GM, WM, CSF) are not suitable for neuroimaging software and tools is available in Table 1 with fine-grain segmentation. The deformation models, multi-atlas their brief descriptions. The list is not exhaustive. The segmentation, model-based segmentation, and machine learning important ones are discussed as follows. models, particularly those based upon CNN, are increasingly being used for fine-grain segmentation of the whole brain or ROIs such as the hippocampus, cerebellum, amygdala, etc. FreeSurfer The theoretical concepts behind various segmentation 19,25,26 27 methodologies have already been detailed in the literature. FreeSurfer is an open-source software suite developed by the The present work focuses on developed models, machine Laboratory of Computational Neuroimaging at the Athinoula learning methods, and tools which have been applied to A. Martinos Center for Biomedical Imaging. The software address the automatic segmentation problem. package is freely available from its website https://surfer.nmr. Table 1. Important Publicly Available Software and Tools for Neuroimaging Studies Software Tools Features FreeSurfer • Free and open source. • Linux and Mac platform. • Analysis and visualization of structural and functional neuroimaging data. • Segmentation uses image intensity and probabilistic atlas with local spatial relationships (atlas- based segmentation). • The current version of FreeSurfer is version 7.0 (May 2020). Statistical Parametric Mapping • Free and open source but requires MATLAB or to use the compiled version. (SPM) • Linux/Unix, Mac, and Windows platform. • The current version of SPM is SPM-12 released in January 2020. • Uses tissue probability maps, segmentation, and labeling functionality further enhanced with toolboxes such as VBM8, CAT12, and AAL3. FMRIB Software Library (FSL) • Free and open source. • Linux and Mac platform. • Analysis tools for FMRI, MRI, and DTI neuroimaging data. • Segmentation is done using FSL-FAST for tissue segmentation and FSL-FIRST for subcortical segmentation (model-based). • The current version of FSL is version 6.0. volBrain • Online MRI brain volumetry system. • Provides volumes of GM, WM, CSF as well as macroscopic areas, also provides subcortical structure segmentation and related volumes. • volBrain web platform also provides additional pipelines like • CERES (CEREbellum Segmentation) uses multi-atlas nonlocal patch-based label fusion • HIPS (HIPpocampus subfield Segmentation) • pBrain (Parkinson related deep nucleus segmentation) Multi-atlas propagation with • Code is available through GitHub. enhanced registration (MAPER) • Uses databases of multiple atlases as the knowledge base. • Requires pincram and FSL for brain extraction and tissue class segmentation. Multi-atlas-based multi-image • Uses multi-atlas-based segmentation algorithm. segmentation (MABMIS) • Uses tree-based group-wise registration of atlas and target images. • Performs the simultaneous segmentation of all available images. • Available at https://www.nitrc.org/projects/mabmis • Last version was released in 2011. Automatic segmentation of hip- • Segmentation of the hippocampus subfield using the included atlas. pocampus subfield (ASHS) • It also allows building own atlas and training it to be used for segmentation. • Can be re-trained and extended to other segmentation. • Free and open source, available for Linux and Mac OS. • Available at https://www.nitrc.org/projects/ashs and last version was released in 2017. Singh and Singh 5 mgh.harvard.edu and is used for a complete range of analysis exists. The tissue classification methodology in SPM considers the of structural and functional imaging data and its visualization. brightness information of voxels along with tissue probability The latest version of the software is version 7.1.0 (released in maps and the position of voxel during classification. The latest May 2020). The segmentation pipeline of FreeSurfer can be version is SPM12 and is last updated in January 2020. The run in a fully automatic manner using the “recon-all all” script. segmentation process of SPM can be further extended using SPM It uses image intensity and probabilistic atlas with local spatial toolboxes; the VBM is a toolbox which has been used in several relationships between subcortical structures for carrying out studies and has now been replaced with the CAT12 toolbox. In a the segmentation. The default pipeline can segment and recent study, it was noticed that CAT12 can contribute better in assign 40 different labels to corresponding voxels in an volumetric analysis than VBM8; this is also on account of automated manner. The segmentation has been extended to normalization and segmentation improvements in SPM12. The further segment 9 amygdala nuclei and 13 hippocampus automated anatomical labeling atlas 3 (AAL3) is another toolbox subfields that have been incorporated in the new statistical of SPM which is a refinement of its previous versions and can atlas based on Bayesian inference build using a postmortem parcellate the brain among 166 labels. specimen at high resolution and has been added in FreeSurfer version 6.0 onward. This allows the automatic simultaneous volBrain (Online Web Platform) segmentation of amygdala nuclei and hippocampus subfields using standard resolution structural MR images. The software volBrain is a web-based pipeline for MRI brain volumetry. is freely available for Linux and Mac platforms, and it provides The pipeline can be accessed from the https://volbrain.upv.es/. graphic as well as command line options. The webserver platform allows researchers to submit their MRI scans in NIFTI format and the online pipeline generates the volumetric measurements which are sent on the registered FMRIB Software Library (FSL) email id after the completion of the job; the same can also be 23,31 FSL is a comprehensive library of neuroimaging tools for downloaded from the volBrain system. The user is required to structural, functional, and diffusion tensor imaging (DTI) register with the volBrain online system and there is a limit on studies. The software has been created and maintained by the number of simultaneous jobs that can be submitted. The FMRIB Analysis Group at the University of Oxford and is entire processing remains a black box from the user’s point of available at https://fsl.fmrib.ox.ac.uk/fsl; the current version view; the theoretical aspects have been described in published of the software is version 6.0. The tissue segmentation is done papers. The volBrain system is primarily based on a multi- using FMRIB Automated Segmentation Tool (FAST), atlas, patch-based segmentation method and utilizes training which uses Markov random field model along with the libraries as implemented in the volBrain online platform. The expectation-maximization algorithm. FAST can be invoked volBrain web platform has also added several brain structure through the command line or GUI in the brain-extracted segmentation methods such as CERES (CEREbellum image volumes. The segmentation pipeline of FSL can also 38 39 Segmentation), HIPS (HIPpocampus Segmentation), and correct RF inhomogeneity and classify the brain among GM, pBrain (Parkinson related deep nucleus segmentation), WM, and CSF tissue types. The probabilistic and partial which are all available from the volBrain platform. volume tissue segmentation used for tissue volume calculation can also be calculated. The subcortical segmentation is Multi-Atlas Propagation With Enhanced performed using FMRIB Integrated Registration and 33 Registration (MAPER) Segmentation Tool (FIRST). FIRST provides model-based segmentation using deformation models. The construction of The multi-atlas propagation with enhanced registration the model was based upon a manually labeled dataset; the (MAPER) is an automatic segmentation tool for structural labels were parameterized as surface meshes modeled as a MRI images into corresponding anatomical sub regions by point distribution model. It first registers the images to 41 using a database of multiple atlases as knowledge base. The MNI152 space by performing affine registration. The “run_ standard MAPER pipeline requires brain extraction and tissue first_all” script does the automatic segmentation of subcortical class segmentation using pincram and FSL-FAST, respectively. structures into 15 different labels. Comparison of Segmentation Accuracy in Statistical Parametric Mapping (SPM) Automatic Methods SPM is a package developed for the analysis of neuroimaging Several studies have been made to judge the relative data coming from several imaging modalities like functional- comparison of various publicly available software tools and MRI, positron emission tomography, magnetoencephalography, algorithms. The FreeSurfer, SPM, and FSL are among the most electroencephalography, etc. This is also a freely available used tools for neuroimaging studies and are not limited to software tool from its website https://www.fil.ion.ucl.ac.uk/spm/ segmentation alone. The other software tools are built for at Wellcome Centre for Human Neuroimaging but requires specific applications. The comparison studies are generally MATLAB platform, even though a compiled version of SPM also 6 Annals of Neurosciences performed on specific datasets with default options and provide Table 2 summarizes the recent comparison of several publicly mixed results which cannot be generalized for all situations. available tools for their accuracy in automatic segmentation. Table 2. Comparison of Some Publicly Available Methods for Automatic Segmentation Research Automatic Seg- Citation mentation Tools Dataset Utilized Conclusion/Findings Yaakub et • MAPER • Hammers_mith brain • Methods applied to the three atlas databases of T1-weighted images. al. • FreeSurfer atlas • Leave-one-out-cross-comparison was done for estimating the seg- (5.3) • Mindboggle-101 da- mentation accuracy of methods. tabase (DKT40 atlas • Both identified known abnormalities in the patient groups. database) • FreeSurfer performed superiorly in AD and Left-HS, whereas MAPER • Atlas created from in the Right-HS dataset. OASIS database for • MAPER performed better in healthy controls. MICCAI 2012 grand challenge Palumbo • SPM-12 • Kirby-21 • GM, WM, subcortical structure segmentation in test-retest MRI data et al. • FreeSurfer • OASIS datasets of healthy volunteers. (6.0) • SPM was found more consistent in the evaluation of ROI volume for intra-method repeatability and inter method reproducibility. Bartel et • FASTSURF • Multicentre phase-III • FASTSURF is a semi-automatic contouring-based segmentation al. • FSL-FIRST trial dataset of SCLC model for the hippocampus and uses a mesh processing technique. • FreeSurfer patients • Comparison of hippocampal atrophy rates was made with manual, • ADNI database FreeSurfer, and FSL. • Semi-automatic FASTSURF model was found superior to compared automatic models. Velasco- • FreeSurfer OASIS dataset • The comparison was made in terms of reproducibility and accuracy Annis et • FSL-First n = 20 (scanned twice), for hippocampus, putamen, thalamus, caudate, pallidum, amygdala, ac- al. • PSTAPLE 1.5T cumbens, and brainstem. (Local MAP • PSTAPLE was found to have superior reproducibility. PSTAPLE) Zandifar • FreeSurfer ADNI database • Applied on hippocampus volumes. et al. 5.3 • All methods show acceptable conformity with manual segmentation. • ANIMAL • Patch-based strategies have a good correlation with manual segmen- • Patch-based tation. methods Perlaki et • FSL-FIRST 30 healthy young Caucasian • Study to compare the segmentation accuracy of the caudate nucleus al. • FreeSurfer subjects and putamen. (v4, 5, and n = 30, 3T • FSL was found to be superior for putamen segmentation. 5.3) Naess- • FreeSurfer • 22 healthy subjects • Thalamus and hippocampus automatic segmentation. Schmidt (5.3) (age 19–40) n = • volBrain (patch-based) provided more accuracy in MP2RAGE images et al. • FSL-FIRST 22, 3T than conventional ones. (4.1.9) • MP2RAGE, for DTI • SPM-12 n = 10 • volBrain Grimm et • FreeSurfer 92 participants in the • Automatic methods are compared with manual segmentation for al. • VBM age range of (18–34, amygdalar and hippocampus volume. mean:21.64) • Both methods were found comparable to manual segmentation. n = 92, 1.5T Fellhauer • FreeSurfer n = 115, 1.5T, MPRAGE • All three detected increase in brain atrophy in AD/MCI group. et al. (5.1) (60 MCI, 34 AD, 32 healthy) • FSL was good with good quality images. • FSL-FAST • SPM was recommended for patient data in difficult measurement (4.1.9) situations. • SPM 8 and 12 (with VBM8) Abbreviations: MCI, mild cognitive impairment Singh and Singh 7 The recent methods of automatic machine learning models Evolving Machine Learning Models for of segmentation are discussed as follows. Automatic Segmentation Several machine learning models have been developed over AssemblyNet the years for the automatic segmentation of brain tissue and This model uses a large assembly of CNNs, each processing anatomical structures of specific structures like the cerebellum different overlapping regions. The framework is arranged in and hippocampus. Fine-grain segmentation of the whole the form of two assemblies of U-Nets, each having 125 3D brain has also been implemented in several methods. A U-Nets (i.e., a total of 250 compact 3D U-Nets). The method summary of some recent machine learning models applied for showed competitive performance in respect of U-Net, patch- automatic brain segmentation is given in Table 3. based joint label fusion, and SLANT-27 methods. The The regular machine learning models do not generalize segmentation accuracy was improved with the use of well and are not suitable for complex imaging modalities; the nonlinearly registered “Atlas prior” for expected classification, deep learning models having multiple layers are increasingly transfer learning to initialize spatially nearest U-Net and being used to address neuroimaging challenges which have multi-scale cascade to communicate between the two benefitted from the advancements in graphical processing assemblies for refinement. The training time was shown to be unit (GPU) processing power. The deep learning models like 7 days in the case of AssemblyNet but the classification time CNN are preferred owing to their application in medical is just 10 min. imaging problems. A typical CNN architecture contains several layers and components such as (a) input layers receiving raw image data, (b) convolutional layers applying SLANT filter (kernel) and producing feature maps, (c) activation The SLANT pipeline first registers the target image to the function layer applying the activation function to the output MNI305 template using affine registration, and this is coming from the convolution layer like rectified linear unit, followed by bias-field correction and normalization. Since (d) pooling layers for downsampling the output of the the whole volume cannot fit into the GPU memory using the preceding layer, and (e) fully connected layer for applying FCN network, the entire MNI space is divided into k weights to feature analysis to predict the label. The typical independent 3D U-Nets. The SLANT-8 divides the volume CNN models are extended in various CNN architectures such 53 54 into eight nonoverlapped subspaces with each subspace of as U-Nets. 3D U-Nets is an extension of traditional U-Net size 86 × 110 × 78 voxels, whereas SLANT-27 divides the architecture for application in 3D biomedical imaging. Figure 2. An Illustration of AssemblyNet Model Using U-Net . Source: Reprinted with permission from Elsevier. 8 Annals of Neurosciences Figure 3. An Illustration of SLANT-27 (27-Network Tiles) Model Using U-Net . (Reprinted with permission from Elsevier). Source: Reprinted with permission from Elsevier. volume into 27 overlapped network tiles with each subspace capacity of the model. The model was successfully applied of 96 × 128 × 88 voxels. The overlapped SLANT-27 thus for 3D whole-brain segmentation and achieved comparable requires majority voting for label fusion. The entire pipeline performance with up to 16 times model compression. is available as a docker image. It provides fine-grain segmentation of the whole brain in more than 100 ROIs. The QuickNAT training time could be as high as 109 h for SLANT-27 on 5,111 training scans using a single GPU (NVIDIA Titan 12 QuickNAT tries to address the limitation of limited manually GB), which can be reduced to 4 h with 27 GPUs, and the curated data for training by first utilizing the existing publicly testing time is roughly 15 min. available software tools such as FreeSurfer to segment the data which is then used to train the network; this pretrained network is then further improved by training with limited 3DQ manually annotated data to achieve high segmentation This is a generalizable method that can be applied to various accuracy. The model does not utilize 3D F-CNN but instead 3D F-CNN architecture such as 3D U-Nets, MALC, and uses three two-dimensional F-CNN, each operating in “V-Net on MALC” for providing model compression of 16 separate planes (i.e., coronal, axial, and sagittal), followed by times to address the memory issues, and it incorporates aggregation resulting in final segmentation and labels quantization mechanism for integrating the trainable scaling corresponding to 27 brain structures. factor and the normalization parameter, which not only The details of other recent methods and architecture have maintains compression but also increases the learning been given in Table 3. Singh and Singh 9 Table 3. Summary of Recent Brain Segmentation Methods Utilizing Machine Learning Models Model/Segmentation Method Area Principle Remarks A. Whole-Brain Segmentation Models for Classifying Among Different Anatomical Labels AssemblyNet CNN/ Utilizes two assemblies of 125 Competitive performance in compari- Brain automatic segmentation 3D U-Nets processing different son with U-Net, joint label fusion, and overlapping brain areas of the SLANT. whole brain. SLANT CNN/ 3D–FCN, addresses memory Training and testing can be optimized by Fine-grain segmentation >100 issues using multiple spatially dis- providing 27 GPUs for SLANT-27 and 8 structures tributed overlapping network tiles for SLANT-8. of U-Nets. QuickNAT CNN/ Fully convolutional and densely Posttraining, the model achieves su- Brain segmentation connected, perior computational performance in (segments 27 structures) pretraining using existing segmen- comparison with patch-based CNN and tation software (FreeSurfer), atlas-based approaches. Also compared fine-tuning to rectify errors using well with FSL and FreeSurfer. manual labels. Bayesian QuickNAT CNN/ F-CNN approach (of QuickNAT) The model has been compared with Brain segmentation with Bayesian inference for seg- QuickNAT and FreeSurfer with manual (segments 33 structures) mentation quality. annotations. 3DQ CNN/ 3D F-CNN with model compres- Integrates training scalable factors and Brain segmentation sion up to 16 times without affect- normalization parameter. (segment 28 structures) ing performance. Useful for storage Increases learning while maintaining critical applications. compression. DeepNAT CNN/ 3D-CNN patch-based model, Uses three CNN layers for pooling, Brain segmentation of 25 the first network removes the normalization, and nonlinearities. structures background, second classifies brain Comparable with other state-of-the-art structure. models. BrainSegNet CNN/ 2D/3D CNN patches Does not require registration, saving on Whole-brain segmentation computational cost. B. Whole-Brain Segmentation Models for Classification Among GM, WM, and CSF Only HyperDense-Net CNN / Fully connected 3D-CNN using Successfully participated in iSEG-2017 Brain segmentation multiple modalities. and MRbrainS-2013 challenge. VoxResNet CNN/ Voxel-wise residual network with Successfully competed in Brain segmentation 25 layers utilizing CNN. MRbrainS-2015 challenge. C. Brain Segmentation Models for Specific Brain Structures (Hippocampus, Cerebellum) HippMapp3r CNN/ CNN architecture based upon Validation is done against FreeSurfer, Hippocampus segmentation U-Net. FSL-First, volBrain, SBHV, and Hip- Initial training on the whole brain, poDeep. Algorithm and trained model the output was trained again with are made publicly available. reduced FOV on the same net- work architecture. CAST CNN/Hippocampus subfield Multi-scale 3D CNN with T1w and Hippocampus subfield segmentation. segmentation T2w imaging modalities as input. ACA-PULCO CNN/ Alternative CNN design using Applied on MPRAGE images. Cerebellum segmentation U-Net with locally constrained optimization. HippoDeep CNN/ Deep learned appearance model The training utilizes multiple cohorts Hippocampus automatic based on CNN. and label derived from FreeSurfer out- segmentation put along with synthetic data. 10 Annals of Neurosciences imaging volumes. Several techniques have been developed to Reliability and Reusability of Automated address the computational cost and memory limitations; for Segmentation Methods example, SLANT-27 breaks the original image volume into 27 overlapping network tiles which can be executed in The automated fine-grain segmentation of the whole brain parallel on 27 GPUs or can run sequentially in the available including specific ROIs has profound usage in neuroscience lesser resources; AssemblyNet goes a step further and uses research. This, however, depends on the quality of MRI two assemblies of 125 3D U-Nets processing different acquisitions, preprocessing, choice of analysis pipelines, and overlapping regions; 3DQ method attempts to provide around many other factors. A retrospective longitudinal study has 16 times compression without affecting performance; other shown a lack of reproducibility arising at the level of methods such as ACA-PULCO and CAST focus on specific acquisition in the MRI scanner and the amount of variability 68 regions such as cerebellum and hippocampus, respectively, being different for different scanners, and this is a cause of rather than whole-brain segmentation. concern having a direct bearing on subsequent analysis Other issues with the machine learning models are with pipelines. The effects of automated pipelines which are often regard to insufficient documentation in the literature; only poorly documented and lack standardization have also been some such methods are publicly made available, even then analyzed in a recent study in the context of functional the training modalities of some of them especially those neuroimaging analysis where the same data were provided to relying on 3D-FCN have a huge computational and memory 70 different research groups, but none of the teams chose bottleneck associated with them and cannot be easily identical pipelines, and the results were also variable across reproduced. There is a need to implement such machine groups, thus emphasizing the need of validation and sharing 69 learning models through computational webservers in a of complex workflows. standardized manner as was the case with volBrain for their effective use and validation. Discussion and Conclusion Acknowledgments The automatic segmentation methods have been quite The authors like to thank Professor Pankaj Seth, National Brain successful in morphometric and volumetric measurements of Research Centre for his guidance, support, and help, including brain tissue and structures in finer details. The publicly proofreading of the manuscript. The support of Dr D. D. Lal, Mr available methods such as FreeSurfer, FSL, SPM, etc. are Sukhkir, and DBT e-Library Consortium (DeLCON) in making sufficiently resilient with respect to noise and artifacts available the research articles is also duly acknowledged. The work introduced at the acquisition stage and have performed has also benefitted from the infrastructure facilities under DIC project consistently across different datasets and are being extensively of BTISNet programme and Dementia Science programme, both used by the neuroimaging community; however, they are still funded by the Department of Biotechnology, Government of India. far from being accepted at par with manual segmentation. Several studies have compared the relative performance of Author Contributions various publicly available methods which have provided MKS conceived the study and was responsible for overall study mixed results and have been discussed in Table 2. direction and planning; KKS provided direction, suggestion, and The machine learning methods particularly those based on supervision. Both the authors discussed the final manuscript. CNN have been shown to perform better than the publicly available software tools, but their performance directly Ethical Statement depends on the amount and type of training data and thus may Not Applicable. not be reproducible across different unseen datasets whose acquisition and protocols vary significantly from the training Declaration of Conflicting Interests data. The insufficient amount of manually labeled training data also affects the performance of such machine learning The authors declared no potential conflicts of interest with respect to models. QuickNAT attempts to improve upon this limitation the research, authorship, and/or publication of this article. by first pretraining the network using auxiliary labels generated using FreeSurfer and then refining the pretrained Funding network with the limited manually labeled data. The authors received no financial support for the research, The 2D F-CNN methods which train the network by using authorship, and/or publication of this article. images slice by slice have an inherent disadvantage of failure to fully utilize the contextual information from neighboring ORCID iDs slices. The 3D F-CNN, on the other hand, faces the memory Mahender Kumar Singh https://orcid.org/0000-0001-9617-6112 limitation of the available GPUs in handling millions of parameters associated with high-resolution clinical MRI Krishna Kumar Singh https://orcid.org/0000-0003-3849-5945 Singh and Singh 11 References tions. Comput Math Methods Med; 2015 ; 2015: 450341. DOI: 10.1155/2015/450341. 1. Driscoll I, Davatzikos C, An Y, et al. Longitudinal pattern of 20. Xue H, Srinivasan L, Jiang S, et al. Automatic segmentation and regional brain volume change differentiates normal aging from reconstruction of the cortex from neonatal MRI. NeuroImage MCI. Neurology 2009; 72: 1906–1913. 2007; 38: 461–477. 2. Franke K and Gaser C. Longitudinal changes in individual 21. Smith SM. Fast robust automated brain extraction. Hum Brain BrainAGE in healthy aging, mild cognitive impairment, and Mapp 2002; 17: 143–155. Alzheimer’s disease. GeroPsych 2012; 25: 235–245. 22. Battaglini M, Smith SM, Brogi S, et al. Enhanced brain 3. Raghuprasad MS and Manivannan M. Volumetric and mor- extraction improves the accuracy of brain atrophy estimation. phometric analysis of pineal and pituitary glands of an Indian NeuroImage 2008; 40: 583–589. inedial subject. Ann Neurosci 2018; 25: 279–288. 23. Jenkinson M, Beckmann CF, Behrens TEJ, et al. FSL. 4. Giorgio A and de Stefano N. Clinical use of brain volumetry. J NeuroImage 2012; 62: 782–790. Magn Reson Imaging 2013; 37: 1–14. 24. Heckemann RA, Ledig C, Gray KR, et al. Brain extraction 5. Wu W-C, Huang C-C, Chung H-W, et al. Hippocampal altera- using label propagation and group agreement: Pincram. PLoS tions in children with temporal lobe epilepsy with or without a ONE 2015; 10: 1–18. history of febrile convulsions: Evaluations with MR volumetry 25. Yazdani S, Yusof R, Karimian A, et al. Image segmentation and proton MR spectroscopy. AJNR Am J Neuroradiol 2005; methods and applications in MRI brain images. IETE Tech Rev 26: 1270–1275. 6. Apostolova LG, Dinov ID, Dutton RA, et al. 3D comparison 2015; 32: 413–427. of hippocampal atrophy in amnestic mild cognitive impairment 26. Iglesias JE and Sabuncu MR. Multi-atlas segmentation of bio- and Alzheimer’s disease. Brain 2006; 129: 2867–2873. medical images: A survey. Med Image Anal 2015; 24: 205–219. 7. Tanskanen P, Veijola JM, Piippo UK, et al. Hippocampus and 27. Fischl B. FreeSurfer. NeuroImage 2012; 62: 774–781. amygdala volumes in schizophrenia and other psychoses in the 28. Fischl B, Salat DH, Busa E, et al. Whole brain segmentation: Northern Finland 1966 birth cohort. Schizophr Res 2005; 75: Automated labeling of neuroanatomical structures in the human 283–294. brain. Neuron 2002; 33: 341–355. 8. Bremner JD, Narayan M, Anderson ER, et al. Hippocampal vol- 29. Saygin ZM, Kliemann D, Iglesias JE, et al. High-resolution ume reduction in major depression. Am J Psychiatry 2000; 157: magnetic resonance imaging reveals nuclei of the human amyg- 115–118. dala: Manual segmentation to automatic atlas. NeuroImage 9. Baxter MG and Murray EA. The amygdala and reward. Nat Rev 2017; 155: 370–382. Neurosci 2002; 3: 563–573. 30. Iglesias JE, Augustinack JC, Nguyen K, et al. A computational 10. Sacchet MD, Livermore EE, Iglesias JE, et al. Subcortical vol- atlas of the hippocampal formation using ex vivo, ultra-high umes differentiate major depressive disorder, bipolar disorder, resolution MRI: Application to adaptive segmentation of in and remitted major depressive disorder. J Psychiatr Res 2015; vivo MRI. NeuroImage 2015; 115: 117–137. 68: 91–98. 31. Smith SM, Jenkinson M, Woolrich MW, et al. Advances in 11. Walker FO. Huntington’s disease. The Lancet 2007; 369: functional and structural MR image analysis and implementa- 218–228. tion as FSL. NeuroImage 2004; 23: S208–S219. DOI: 10.1016/j. 12. Hokama H, Shenton ME, Nestor PG, et al. Caudate, putamen, neuroimage.2004.07.051. and globus pallidus volume in schizophrenia: A quantitative 32. Zhang Y, Brady M, and Smith S. Segmentation of brain MR MRI study. Psychiatry Res Neuroimaging 1995; 61: 209–229. images through a hidden Markov random field model and the 13. Sterling NW, Du G, Lewis MM, et al. Striatal shape in expectation-maximization algorithm. IEEE Trans Med Imaging Parkinson’s disease. Neurobiol Aging 2013; 34: 2510–2516. 2001; 20: 45–57. 14. Shen L, Firpi HA, Saykin AJ, et al. Parametric surface model- 33. Patenaude B, Smith SM, Kennedy DN, et al. A Bayesian model ing and registration for comparison of manual and automated of shape and appearance for subcortical brain segmentation. segmentation of the hippocampus. Hippocampus 2009; 19: NeuroImage 2011; 56: 907–922. DOI: 10.1016/j.neuroim- 588–595. age.2011.02.046. 15. Cherbuin N, Anstey KJ, Réglade-Meslin C, et al. In vivo hip- 34. Ashburner J. SPM: A history. NeuroImage 2012; 62: 791–800. pocampal measurement and memory: A comparison of manual 35. Farokhian F, Beheshti I, Sone D, et al. Comparing CAT12 and tracing and automated segmentation in a large community- VBM8 for detecting brain morphological abnormalities in tem- based sample. PLoS ONE 2009; 4: e5265. poral lobe epilepsy. Front Neurol 2017; 8: 1–7. 16. Wenger E, Mårtensson J, Noack H, et al. Comparing man- 36. Rolls ET, Huang CC, Lin CP, et al. Automated anatomical ual and automatic segmentation of hippocampal volumes: Reliability and validity issues in younger and older brains. Hum labelling atlas 3. NeuroImage 2020; 206: 116189. Brain Mapp 2014; 35: 4236–4248. 37. Manjón JV and Coupé P. volBrain: An online MRI brain volu- 17. Gispert JD, Reig S, Pascau J, et al. Method for bias field cor- metry system. Front Neuroinform 2016; 10: 30. DOI: 10.3389/ rection of brain T1-weighted magnetic images minimizing seg- fninf.2016.00030. mentation error. Hum Brain Mapp 2004; 22: 133–144. 38. Romero JE, Coupé P, Giraud R, et al. CERES: A new cerebellum 18. Song S, Zheng Y, and He Y. A review of methods for bias cor- lobule segmentation method. NeuroImage 2017; 147: 916–924. rection in medical images. Biomed Eng Rev 2017; 3: 1–10. 39. Romero JE, Coupé P, and Manjón J v. HIPS: A new hippo- 19. Despotović I, Goossens B, and Philips W. MRI segmenta- campus subfield segmentation method. NeuroImage 2017; 163: tion of the human brain: Challenges, methods, and applica- 286–295. 12 Annals of Neurosciences tation. In: Ourselin S, Joskowicz L, Sabuncu MR, et al., 40. Manjón J v, Bertó A, Romero JE, et al. pBrain: A novel pipeline eds. Medical Image Computing and Computer-Assisted for Parkinson related brain structure segmentation. NeuroImage Intervention – MICCAI 2016. Springer International Clin 2020; 25: 102184. Publishing, 2016: pp. 424–432. 41. Heckemann RA, Keihaninejad S, Aljabar P, et al. Improving 55. Coupé P, Mansencal B, Clément M, et al. AssemblyNet: A intersubject image registration using tissue-class information large ensemble of CNNs for 3D whole brain MRI segmenta- benefits robustness and accuracy of multi-atlas based anatomi- tion. NeuroImage 2020; 219: 117026. cal segmentation. NeuroImage 2010; 51: 221–227. 56. Huo Y, Xu Z, Xiong Y, et al. 3D whole brain segmentation 42. Jia H, Yap P-T, and Shen D. Iterative multi-atlas-based multi- using spatially localized atlas network tiles. NeuroImage 2019; image segmentation with tree-based registration. NeuroImage 194: 105–119. 2012; 59: 422–430. 57. Paschali M, Gasperini S, Roy AG, et al. 3DQ: Compact quan- 43. Yushkevich PA, Pluta JB, Wang H, et al. Automated volumetry tized neural networks for volumetric whole brain segmenta- and regional thickness analysis of hippocampal subfields and tion. In Shen D. et al. (eds.) Medical Image Computing and medial temporal cortical structures in mild cognitive impair- Computer Assisted Intervention - MICCAI 2019. MICCAI ment. Hum Brain Mapp 2015; 36: 258–287. 2019. Lecture Notes in Computer Science, vol 11766. Springer, 44. Yaakub SN, Heckemann RA, Keller SS, et al. On brain atlas Cham, 2019: pp. 438–446. choice and automatic segmentation methods: a comparison 58. Guha Roy A, Conjeti S, Navab N, et al. QuickNAT: A fully of MAPER & FreeSurfer using three atlas databases. Sci Rep convolutional network for quick and accurate segmentation of 2020; 10: 2837. neuroanatomy. NeuroImage 2019; 186: 713–727. 45. Palumbo L, Bosco P, Fantacci ME, et al. Evaluation of the intra- 59. Roy AG, Conjeti S, Navab N, et al. Bayesian QuickNAT: Model and inter-method agreement of brain MRI segmentation soft- uncertainty in deep whole brain segmentation for structure-wise ware packages: A comparison between SPM12 and FreeSurfer quality control. NeuroImage 2019; 195: 11–22. v6.0. Physica Medica 2019; 64: 261–272. 60. Wachinger C, Reuter M, and Klein T. DeepNAT: Deep con- 46. Bartel F, Vrenken H, van Herk M, et al. Fast segmentation volutional neural network for segmenting neuroanatomy. through surface fairing (FastSURF): A novel semi-automatic NeuroImage 2018; 170: 434–445. hippocampus segmentation method. PLoS ONE 2019; 14: 61. Mehta R, Majumdar A, and Sivaswamy J. BrainSegNet: A con- e0210641. volutional neural network architecture for automated segmenta- 47. Velasco-Annis C, Akhondi-Asl A, Stamm A, et al. tion of human brain structures. J Med Imaging 2017; 4: 24003. Reproducibility of brain MRI segmentation algorithms: 62. Dolz J, Gopinath K, Yuan J, et al. HyperDense-Net: A hyper- Empirical comparison of local MAP PSTAPLE, FreeSurfer, densely connected CNN for multi-modal image segmentation. and FSL-FIRST. J Neuroimaging 2018; 28: 162–172. IEEE Trans Med Imaging 2019; 38: 1116–1126. 48. Zandifar A, Fonov V, Coupé P, et al. A comparison of accurate 63. Chen H, Dou Q, Yu L, et al. VoxResNet: Deep voxelwise automatic hippocampal segmentation methods. NeuroImage residual networks for brain segmentation from 3D MR images. 2017; 155: 383–393. NeuroImage 2018; 170: 446–455. 49. Perlaki G, Horvath R, Nagy SA, et al. Comparison of accuracy 64. Goubran M, Ntiri EE, Akhavein H, et al. Hippocampal segmen- between FSL’s FIRST and Freesurfer for caudate nucleus and tation for brains with extensive atrophy using three-dimensional putamen segmentation. Sci Rep 2017; 7: 2418. convolutional neural networks. Hum Brain Mapp 2020; 41: 50. Næss-Schmidt E, Tietze A, Blicher JU, et al. Automatic 291–308. thalamus and hippocampus segmentation from MP2RAGE: 65. Yang Z, Zhuang X, Mishra V, et al. CAST: A multi-scale con- Comparison of publicly available methods and implications for volutional neural network based automated hippocampal sub- DTI quantification. Int Comput Assist Radiol Surg 2016; 11: field segmentation toolbox: CAST: Segmenting hippocampal 1979–1991. subfields. NeuroImage 2020; 218: 116947. 51. Grimm O, Pohlack S, Cacciaglia R, et al. Amygdalar and hippo- 66. Han S, Carass A, He Y, et al. Automatic cerebellum anatomical campal volume: A comparison between manual segmentation, parcellation using U-Net with locally constrained optimization. Freesurfer and VBM. J Neurosci Methods 2015; 253: 254–261. NeuroImage 2020; 218: 116819. 52. Fellhauer I, Zöllner FG, Schröder J, et al. Comparison of auto- 67. Thyreau B, Sato K, Fukuda H, et al. Segmentation of the hippo- mated brain segmentation using a brain phantom and patients campus by transferring algorithmic knowledge for large cohort with early Alzheimer’s dementia or mild cognitive impairment. processing. Med Image Anal 2018; 43: 214–228. Psychiatry Res Neuroimaging 2015; 233: 299–305. 68. Yang CY, Liu HM, Chen SK, et al. Reproducibility of brain 53. Ronneberger O, Fischer P, and Brox T. U-Net: Convolutional morphometry from short-term repeat clinical MRI examina- networks for biomedical image segmentation. In: Navab N, tions: A retrospective study. PLoS ONE 2016; 11: e0146913. Hornegger J, Wells WM, et al., eds. Medical Image Computing 69. Botvinik-Nezer R, Holzmeister F, Camerer CF, et al. Variability and Computer-Assisted Intervention – MICCAI 2015. Springer in the analysis of a single neuroimaging dataset by many teams. International Publishing, 2015: pp. 234–241. 54. Çiçek Ö, Abdulkadir A, Lienkamp SS, et al. 3D U-Net: Nature 2020; 582: 84–88. Learning dense volumetric segmentation from sparse anno-

Journal

Annals of NeurosciencesSAGE

Published: Jan 1, 2021

There are no references for this article.