Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

MRI Super-Resolution with Ensemble Learning and Complementary Priors

MRI Super-Resolution with Ensemble Learning and Complementary Priors > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 1 MRI Super-Resolution with Ensemble Learning and Complementary Priors Qing Lyu, Hongming Shan, and Ge Wang, Fellow, IEEE protocol [1], [2]. Abstract—Magnetic resonance imaging (MRI) is a widely used SR has been a hot topic in the computational vision field over medical imaging modality. However, due to the limitations in the past decades. A large number of methods were proposed to hardware, scan time, and throughput, it is often clinically improve image resolution retrospectively. Roughly speaking, challenging to obtain high-quality MR images. The super- super-resolution algorithms can be divided into the two resolution approach is potentially promising to improve MR image categories: model-based and learning-based. Interpolation quality without any hardware upgrade. In this paper, we propose algorithms [3], [4], like bilinear, bicubic, and nearest neighbor an ensemble learning and deep learning framework for MR image super-resolution. In our study, we first enlarged low resolution interpolation techniques, are representatives of simple model- images using 5 commonly used super-resolution algorithms and based approaches, which can be directly used to enlarge images. obtained differentially enlarged image datasets with Wiener filtering and iterative deblurring algorithms utilize complementary priors. Then, a generative adversarial network knowledge on the system point spread function (PSF) to recover (GAN) is trained with each dataset to generate super-resolution image resolution [5], [6], and are also considered as model- MR images. Finally, a convolutional neural network is used for based algorithms. Dictionary learning-based super-resolution ensemble learning that synergizes the outputs of GANs into the final MR super-resolution images. According to our results, the techniques [7]-[9] are examples of learning-based algorithms. ensemble learning results outcome any one of GAN outputs. Recently, with the rapid evolution of machine learning, Compared with some state-of-the-art deep learning-based super- especially deep learning, deep neural networks have become resolution methods, our approach is advantageous in suppressing popular among SR studies. In this context, diverse neural artifacts and keeping more image details. network architectures were designed and tested. Successful architectures include convolutional neural networks (CNNs) Index Terms— Deep learning, ensemble learning, generative adversarial network (GAN), magnetic resonance imaging (MRI), [10]-[16], generative adversarial networks (GANs) [17]-[20], super-resolution residual neural networks (ResNets) [21]-[24], and recurrent neural networks (RNNs) [25]-[27]. Specifically, for MRI SR studies, models based on CNN, ResNet, and GAN were I. INTRODUCTION reported [28]-[32]. Although there were a plenty of decent SR results in the literature, bothersome artifacts or distortions were agnetic resonance imaging (MRI) is one of the most M often observed in the SR results produced by these methods. commonly used medical imaging modalities over the Therefore, the challenge remains for achieving superior SR on world. It noninvasively depicts structural and functional MR images. In other words, MRI SR research must deliver features inside a patient in rich contrasts. Compared with other images with faithful details and little artifacts in order to be medical imaging modalities such as computed tomography clinically relevant. Since the existing methods cannot perfectly (CT) and nuclear imaging, MRI does not involve ionizing satisfy such a requirement, it is necessary to develop new SR radiation. Furthermore, MRI plays a dominating role in methods based on all encouraging results. neurological imaging especially brain research. Among important deep learning strategies, ensemble deep However, MRI suffers from major physical limitations in learning is a meta-method for further enhancement of deep background magnetic field, gradient fields, imaging speed, and learning results [33], [34]. By combining multiple models, signal-to-noise ratio. Hence, the pixel size of a clinical MR ensemble learning is expected to produce synergistic results image is often constrained to an order of millimeters. To obtain than the results from individual models. Merits of ensemble higher resolution MR images, either a more complicated MRI learning were successfully demonstrated to improve image system with stronger background magnetic field or a longer quality [35], [36]. In this MRI SR study, by combining multiple scan time needs to be used, which will dramatically increase GAN models trained on different image prior datasets, we expenses and is rarely clinically applicable. To address this demonstrate better MRI SR image quality than that associated problem, the super-resolution (SR) approach holds a great with any single model. In this paper, we describe a promise that needs to change neither hardware nor scanning (Corresponding author: Ge Wang) All authors are with the Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, 12180, USA (email: [email protected], [email protected], [email protected]) > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 2 Fig. 1. Overall workflow of the SR method proposed in this paper. convolutional neural network to integrate diverse SR results datasets, each of which corresponds to one of the five SR from individual models. Our method produces better image algorithms. These image datasets are separately fed into the quality than state-of-the-art SR models, being capable of corresponding GAN for training and then to produce five SR detailing textures and avoiding artifacts. counterparts. The final step is to use a CNN to do the ensemble learning and obtain the final MRI SR result. II. METHODOLOGY A. Down-sampling Differing from commonly used down-sampling strategy that 𝑝 ×𝑞 Assuming that 𝐼 ∈ ℝ is a low-resolution (LR) MR 𝐿𝑅 down samples data in the image domain, we chose to down- 𝑚 ×𝑛 image of the size 𝑝 × 𝑞 and 𝐼 ∈ ℝ the corresponding 𝐻𝑅 sample data in the frequency domain. The original HR images high-resolution (HR) MR image of the size 𝑚 × 𝑛 . The were converted into the frequency domain via the 2D Fourier relationship between them can be expressed as transform. Only the central 25% of the data points were kept in the k-space, while all peripheral data points were zeroed out. 𝐼 = 𝑓 (𝐼 ) (1) Then, the cropped dataset was converted back into an image 𝐿𝑅 𝐻𝑅 through the 2D inverse Fourier transform as a LR image. Using 𝑚 ×𝑛 𝑝 ×𝑞 this down-sampling method, we obtained LR images with 2- where 𝑓 : 𝐼 ∈ ℝ → 𝐼 ∈ ℝ denotes the down- 𝐿𝑅 fold resolution degradation, which means 𝑝 = 𝑚 /2 and 𝑞 = sampling/blurring process that creates a LR counterpart from a 𝑛 /2. HR image. The overall SR process is to implement an −1 approximate inverse function 𝑔 ≈ 𝑓 B. Processing Before LR images are fed into the neural network, we adopt −1 𝑔 ≈ 𝑓 : 𝐼 ≈ 𝑔 (𝐼 ) (2) 𝐻𝑅 𝐿𝑅 existing SR algorithms to enlarge a LR image to the size of the target HR image. This processing step was also used in some As an ill-posed problem, it is impossible to find this inverse SR studies like VDSR [13] and SRCNN [11]. With such a function exactly. Instead, priors can be used to acquire an processing step, the quality of enlarged images can be somehow −1 approximation of 𝑓 . Multiple types of priors are used in this improved so that it is easier for neural networks to extract image −1 study to approximate 𝑓 as closely to the ideal function as features and obtain better prediction results. Moreover, we can possible. establish datasets with complementary priors through using Fig. 1 presents the proposed overall workflow of SR method. multiple SR algorithms in this processing step. In this paper, in In our scheme, we first down-sample original HR images and order to distinguish enlarged images with the latter neural obtain LR counterparts, and then use five different model-based network results, we call enlarged images “processed LR images SR algorithms to process LR images and obtain five image (PLR)” rather than SR images. In this work, 5 classic SR 𝐻𝑅 > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 3 Fig. 2. The architecture of the proposed GAN. (a) The generator and (b) the discriminator. algorithms were used to enlarge LR images and obtain the and HR images, adding such skip connections can greatly boost corresponding PLRs, which are zero interpolation filling (ZIP) the training process and keep the generated images structurally [37], bicubic interpolation (BI) [3], new edge directed similar to the input LR images. Similarly, the discriminator interpolation (NEDI) [4], sparse coding (SC) [7], and adjusted consists of five blocks followed by a global average pooling anchored neighborhood regression (A+) [9]. ZIP is a simplest (GAP) layer. Each block includes convolution, layer super-resolution method that adds zeros in the peripheral region normalization, and leaky-ReLU (LReLU). Note that we replace in the frequency domain. It is widely used to enlarge MR the fully-connected layer with a GAP layer to reduce the images. BI and NEDI are two improved interpolation-based amount of parameters in the neural network and facilitate the algorithms, while SC and A+ are two dictionary-learning-based training process [44]. Each block in the discriminator performs algorithms. In the A+ algorithm, 1,024 dictionary atoms were convolution, layer normalization and ReLU activation function used, which were obtained from a training dataset of 0.5 million similar to what the block in the generator does. The details of natural images. Similarly, the SC method also used a dictionary the generator and the discriminator are listed in Table I and II. contained 1,024 atoms. Using these different SR algorithms, The objective function of the generator is as follows: different enlarged images were created, each with its unique priors. Next, we used these enlarged images as the inputs to (4) min ℒ = ℒ + 𝜆 ℒ + 𝜆 ℒ + 𝜆 ℒ 𝐺 𝑑𝑣𝑎 1 𝑟𝑎𝑔 2 𝑠𝑒𝑚 3 𝑝𝑟𝑒 train the corresponding GAN models separately. Each ℒ = −𝔼 [𝐷 (𝐺 (𝐼 ))] (5) 𝑑𝑣𝑎 𝐼 𝑃𝐿𝑅 enlarging/deblurring process can be expressed as 𝑅𝐿𝑃 ℒ = 𝔼 ‖∇(𝐺 (𝐼 )) − ∇(𝐼 )‖ (6) 𝑟𝑎𝑔 (𝐼 ,𝐼 ) 𝑃𝐿𝑅 𝐻𝑅 𝑅𝐿𝑃 𝐻𝑅 𝐼 = ℎ (𝐼 ) (3) 𝑃𝐿𝑅 ,𝑖 𝑖 𝐿𝑅 ℒ = 𝔼 ‖∅(𝐺 (𝐼 )) − ∅(𝐼 )‖ (7) 𝑝𝑟𝑒 (𝐼 ,𝐼 ) 𝑃𝐿𝑅 𝐻𝑅 𝑅𝐿𝑃 𝐻𝑅 𝑚 ×𝑛 where 𝐼 ∈ ℝ is a deblurred/enlarged image. ℎ : 𝐼 ∈ 𝑃𝐿𝑅 ,𝑖 𝑖 𝐿𝑅 𝑝 ×𝑞 𝑚 ×𝑛 ℝ → 𝐼 ∈ ℝ denotes the processing process. As 𝑃𝐿𝑅 ,𝑖 where 𝜃 stands for the generator parameters. ℒ is the 𝐺 𝑑𝑣𝑎 there were 5 SR algorithms used in this processing step, 𝑖 = adversarial loss. ℒ is the gradient loss, which is the L2 norm 𝑟𝑎𝑔 1,2, … ,5. difference between gradients of the generated image and its C. GAN model ground truth. ∇ stands for the gradient calculated in both x and y axis. ℒ and ℒ are the mean squared error (MSE) and The GAN model with Wasserstein distance and gradient 𝑠𝑒𝑚 𝑝𝑟𝑒 penalty (WGAN-GP) [38] is selected in this study. The the perceptual loss respectively. MSE reflects the image content generator used in this study is a modified version of the similarity between generated output images and processed input encoder-decoder that was originally used for CT denoising [39]. images at the pixel level, while the perceptual loss shows the The architectures of the generator and the discriminator are similarity in a high-level feature space. We use the pretrained shown in Fig. 2. For the generator, it contains seven blocks with VGG19 model to calculate the perceptual loss [39], [40], and ∅ th three skip connections. Each block involves convolution, layer represents the feature map of the 16 convolutional layer in the normalization (LN), and ReLU activation operations VGG19 model. As the mean-squared-error loss tends to blur sequentially. As mentioned in [38], batch normalization cannot images, the gradient loss is introduced to penalize any blurring be used in the WGAP-GP model, instead, we utilize layer and enhance the edge. The objective function of the normalization. With skip connections, feature maps in the discriminator is the standard WGAN-GP equation shown in previous blocks are concatenated to the feature maps in the [38]. Overall, this GAN SR process is then expressed as latter blocks. Because of the structural similarity between LR > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 4 𝐼 = 𝐺 (𝐼 ) = 𝜑 (𝐼 , 𝜃 ) (8) A. Experimental implementation and neural network ,𝑖 𝑃𝐿𝑅 ,𝑖 𝑃𝐿𝑅 ,𝑖 𝐺 ,𝑖 convergence 𝑚 ×𝑛 where 𝐼 ∈ ℝ is a output image from generator in GAN, The NYU fastMRI dataset was used in this study [41]. All ,𝑖 𝑚 ×𝑛 𝑚 ×𝑛 images were reconstructed from proton density weighted knee 𝜑 : 𝐼 ∈ ℝ → 𝐼 ∈ ℝ denotes a GAN deblurring 𝑃𝐿𝑅 ,𝑖 ,𝑖 scans under 1.5 or 3 Tesla. The size of original HR images is process for an SR image, 𝜃 represents the parameters of a 𝐺 ,𝑖 320 × 320. Totally, 159 patient scans with 5,744 slices were GAN model. As there are 5 SR algorithms used in this step, 𝑖 = used for training (80%) and validation (20%), and additional 43 1,2, … ,5. patient scans with 1,500 slices for testing. Before training, each processed image was decomposed into forty-nine 80 × 80 TABLE I STRUCTURAL DETAILS OF THE GENERATOR, WHERE n1s1 MEANS patches. During the training process of the network models, 1 CONVOLUTION KERNELS WITH STRIDE OF 1. weights in the objective function were selected to be 0.1, 0.1, Block1 Block2 Block3 Block4 Block5 Block6 Block7 and 1 for 𝜆 , 𝜆 , and 𝜆 respectively via the grid searching 1 2 3 n32s1 n64s1 n128s1 n256s1 n128s1 n64s1 n32s1 strategy. The training continued for 50 epochs with the learning n32s1 n64s1 n128s1 n256s1 n128s1 n64s1 n1s1 -5 rate of 2×10 and batch size of 64 for each GAN model. During the training process, to visualize the convergence of the TABLE II STRUCTURAL DETAILS OF THE DISCRIMINATOR, WHERE n1s1 involved models, the values of MSE and perceptual loss are MEANS 1 CONVOLUTION KERNELS WITH STRIDE OF 1. recorded and showed in Fig. 3. For ensemble learning, the CNN Block1 Block2 Block3 Block4 Block5 -5 was trained over 80 epochs with the learning rate of 2×10 and n64s2 n128s2 n256s2 n512s1 n1s1 batch size of 4. The experiment was conducted in Tensorflow on a single GTX 1080Ti GPU. According to the MSE and perceptual loss curves in Fig. 3, D. Ensemble learning all the models converge well during the training process since Ensemble learning is to synergize diverse information all the curves become gradually steady with the number of sources for the best results. In this study, SR images training epochs. Compared with the MSE curves, it can be individually obtained can be integrated via ensemble learning found that the curve of A+ is lower than the other four curves, to improve SR MR image quality. Specifically, the final SR which shows that the A+ based GAN model can achieve the results are derived from different types of GAN SR predictions. best SR image with the highest similarity at the pixel level. In In the processing step, we utilized the five SR algorithms to the high-level space, as extracted by the perceptual loss, both process LR images and obtain five sets of enlarged/deblurred the A+ and ZIP curves show the lowest values while the NEDI images. Then, we train five GAN models (with the same curve is the highest. However, all the curves are pretty close to structure) separately corresponding to each of these five each other after 40 epochs. algorithms and then obtain five sets of prediction results. After that, a CNN model is used to integrate all these images through ensemble learning. The structure of the CNN is the same as the generator in the proposed GAN. Images from each type of prediction results are concatenated before input to the CNN for training, validation, and testing. The mean-absolute-error (MAE) is used in the loss function during the training process. The ensemble learning process can be expressed as 𝐼 = 𝜙 (∑ 𝐼 , Θ) (9) ,𝑖 Fig. 3. Comparisons between the involved model types during the training process in terms of (a) MSE and (b) perceptual loss. 𝑚 ×𝑛 where 𝐼 ∈ ℝ denotes ensemble learning results which are 𝑚 ×𝑛 𝑚 ×𝑛 also our SR outcomes, 𝜙 : 𝐼 ∈ ℝ → 𝐼 ∈ ℝ denotes ,𝑖 B. Processed results the ensemble learning process, Θ represents the parameters of It is shown in Fig. 4-6 that PLR images had smoother edges and the CNN model. clearer shapes compared with the corresponding LR images. As different SR algorithms were used in the processing step, PLR III. RESULTS images obtained from these SR algorithms were somehow Our ensemble learning SR approach was streamlined and different. As far as the edges in Fig. 5(a) are concerned, it can evaluated for MRI SR. It produced SR images with more be found that all the PLR images give clearer edges than the structural details, less artifacts and distortion. original LR image. Among all the 5 processed images, ZIP and In the following, we will show our processed images, GAN A+ images are closer to the HR image than the other three predictions, and ensemble learning results, and compare our SR image types. However, these PLR images also introduced some results with state-of-the-art methods. To evaluate image quality, artifacts and distortions. Some edges in the NEDI PLR image the metrics we used include structural similarity (SSIM) and were slightly distorted, with ring or streak artifacts in the ZIP peak signal-to-noise ratio (PSNR). 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 5 Fig. 4. Input, intermediate and output images in the SR workflow in (a) Cases 1 and (b) 2, where processed results are in the yellow boxes, LR images are in the purple boxes, HR images in the green boxes, ensemble learning results in the red boxes, and GAN predictions in the orange boxes. For the processed results and GAN predictions from left to right are ZIP, BI, NEDI, SC, A+. White dotted boxes are zoomed in region shown in Fig. 5 and 6. Fig. 5. Zoomed in results of regions marked by white dotted boxes in Cases 1, where processed results are in the yellow boxes, LR images are in the purple boxes, HR images in the green boxes, ensemble learning results in the red boxes, and GAN predictions in the orange boxes. For the processed results and GAN predictions from left to right are ZIP, BI, NEDI, SC, A+. > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 6 Fig. 6. Zoomed in results of regions marked by white dotted boxes in Cases 2, where processed results are in the yellow boxes, LR images are in the purple boxes, HR images in the green boxes, ensemble learning results in the red boxes, and GAN predictions in the orange boxes. For the processed results and GAN predictions from left to right are ZIP, BI, NEDI, SC, A+. TABLE III QUANTITATIVE EVALUATION OF THE PROCESSED IMAGES, GAN PREDICTIONS AND ENSEMBLE LEARNING RESULTS IN FIG. 5 AND 6. Preprocessed results GAN predictions Ensemble ZIP BI NEDI SC A+ ZIP BI NEDI SC A+ learning Figure SSIM 0.86 0.81 0.84 0.80 0.86 0.88 0.88 0.87 0.87 0.87 0.95 4(a) PSNR 25.82 25.92 23.53 24.22 24.52 26.67 28.10 26.07 27.71 27.41 27.61 SSIM 0.86 0.82 0.84 0.80 0.86 0.88 0.88 0.86 0.87 0.87 0.95 Figure 4(b) PSNR 28.05 26.02 26.69 25.86 28.36 28.52 28.03 26.79 28.04 27.43 32.67 Figure SSIM 0.85 0.79 0.82 0.77 0.85 0.86 0.86 0.85 0.86 0.86 0.92 5(a) PSNR 25.23 23.49 24.96 23.92 25.95 26.90 26.73 25.74 26.34 27.15 29.44 Figure SSIM 0.71 0.62 0.66 0.61 0.71 0.70 0.71 0.68 0.69 0.71 0.82 5(b) PSNR 24.20 22.57 23.65 21.87 24.28 24.25 24.55 24.10 24.22 24.37 25.98 and SC PLR images. In Fig. 5(b), for each PLR image the are also presented in the GAN SR predictions. As the GANs texture details became smoother while holes in the peripheral were trained on different training datasets, when GAN SR region were more distinguishable compared with the LR image. predictions are compared, differences among them are clearly Despite some artifacts introduced, the SC image contains the detectable for the same ground truth. For example, when clearest hole shapes. Each algorithm had its own strengths and comparing edges pointed by the red arrow in Fig. 5(a), it can be weaknesses. After the processing step, we obtained diverse found that shapes of edges in the GAN SR predictions were enlarged/deblurred images, some of which present edges well slightly different. Another example was shown in Fig. 6(a), while the others of which preserve details satisfactorily. shapes of the hole pointed by the red arrows were moderately different from each other. These shape deformations can be C. Super-resolution results through GAN called distortions. The quality of the prediction images was After training the GAN models separately, five types of GAN quantified in Table III. It can be seen in the table that despite SR prediction results were obtained, as shown in each purple the image quality seems visually improved in those SR box in in Fig. 3. Compared with the PLR results, the image prediction results, the PSNR and SSIM values of the GAN quality of GAN SR predictions was improved. Generally, they predictions were just close to or slightly higher than the PLR included more details, clearer edges, and less blurs, which are results. The appearance of artifacts and distortions in the SR much closer to the original HL images than the PLR versions. predictions contributed to compromise the PSNR and SSIM For GAN SR predictions, edges in Fig. 5(a) become much values. After training, the GAN SR predictions produced clearer while textures in Fig. 6(b) are less blurred compared to numerous tiny artifacts and distortions. For example, the SSIM the corresponding edges and textures in PLR images. However, value of the ZIP GAN prediction was even lower than the just as shown in the PLR images, some artifacts and distortions > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 7 corresponding PLR image. When the two images are compared, used for the ensemble learning, denoted as 3 CNN. Moreover, despite textures in the GAN SR prediction image were clearer simple averaging algorithm was used to replace the CNN and closer to the HR image, there were streak artifacts model. According to the amount of GAN SR predictions used, produced. As a result, the SSIM value was decreased after they are referred as 3 averaging and 5 averaging. Representative training. results are presented in Fig. 7 and Table IV. It can be found that the combination of 3 types of GAN SR predictions yielded less D. Super-resolution results through ensemble learning PSNR and SSIM values than the integration of 5 types of GAN Compared with GAN SR predictions, the most distinct SR predictions. Importantly, there were more artifacts characteristic of the ensemble learning results is that artifacts noticeable in the combination of the 3 types of ensemble and noise were greatly reduced as shown in Fig. 4-6. Also, the learning results, as pointed by the red arrows in Fig. 7. All these ensemble learning results showed richer textural details than results demonstrate that ensemble learning results can be any GAN SR predictions, for example, when comparing the further improved by combining more GAN SR predictions. regions pointed by the red arrows in Fig. 5(b), it can be found When comparing the result from simple averaging followed by that the middle peak can be well presented after ensemble CNN-based integration, the use of CNN for ensemble learning learning. In addition, ensemble learning essentially avoided edge distortions, such as the edges pointed by the red arrows in Fig. 5(a) were very close to the ground truth. Quantitatively, the ensemble learning results achieved the highest PSNR and SSIM values among all the results, indicating that ensemble learning greatly improved the image quality. Fig. 8. Comparison of the proposed method and state-of-the-art methods. TABLE V QUANTITATIVE EVALUATION OF THE RESULTS IN FIG. 8. FSRCNN LapSRN ESPCN VDSR Ours SSIM 0.88 0.88 0.81 0.89 0.95 Case1 PSNR 25.90 25.68 21.14 26.96 27.61 SSIM 0.88 0.88 0.86 0.89 0.95 Case2 PSNR 28.33 28.63 24.51 28.25 32.67 SSIM 0.86 0.86 0.83 0.88 0.92 Case3 PSNR 25.58 25.22 21.57 26.12 29.44 Fig. 7. Ensemble learning results in Cases 1-4, where k averaging means that SSIM 0.70 0.71 0.66 0.71 0.82 the result is simple average of k GAN prediction results, and k CNN means that Case4 PSNR 24.04 23.94 21.06 24.11 25.98 ensemble learning is done through the CNN combining k prediction results. TABLE IV QUANTITATIVE EVALUATION OF THE RESULTS IN FIG. 7. 3 average 3 CNN 5 average 5 CNN SSIM 0.90 0.91 0.90 0.95 Case1 PSNR 26.20 28.24 26.45 27.61 SSIM 0.89 0.91 0.90 0.95 Case2 PSNR 28.38 28.90 28.72 32.67 SSIM 0.86 0.88 0.87 0.92 Case3 PSNR 25.86 26.23 25.97 29.44 SSIM 0.69 0.70 0.71 0.82 Case4 PSNR 23.75 24.20 23.95 25.98 To investigate how much improvement can be made by the ensemble learning, we further compared our results from ensemble learning with different methods. Instead combining 5 types of GAN SR predictions for ensemble learning, only three types of GAN SR predictions were used. Only GAN SR Fig. 9. Pixel intensity accuracy of the proposed method and state-of-the-art predictions trained from ZIP, BI and NEDI PLR images were methods. > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 8 can achieve better results, showing that optimal integration of In the processing step, 5 different SR algorithms were used diverse SR style would improve image quality further. to enlarge LR images. As a result, 5 types of enlarged PLR image datasets were obtained, each with its unique image priors. From Fig. 4-6, it can be found that each SR algorithm TABLE VI can somehow closely preserve some part of image features. For COMPARISON OF OUR METHOD AND OTHER STATE-OF-THE-ART example, ZIP, BC, and NEDI algorithms are good at presenting METHODS (MEAN±STD). edges with clear and smooth shapes, SC and A+ PLR images FSRCNN LapSRN ESPCN VDSR Ours show texture details. By combining all these SR algorithms SSIM 0.87±0.02 0.88±0.02 0.83±0.03 0.88±0.02 0.92±0.01 PSNR 28.87±2.33 28.80±2.38 24.93±3.30 28.99±2.25 31.35±2.15 together, we can create PLR datasets that are with complementary image priors and finally obtain good SR results that are very close to the ground truth after ensemble learning. Fig. 7 shows the necessity of combining all priors. When only E. Comparison with other deep learning-based methods ZIP and interpolation-based methods were used, ensemble Fig. 8-9 and Table V-VI compare our results with the learning results would not well present texture details. For counterparts obtained using other state-of-the-art SR methods. example, the middle peak in the second row in Fig. 7 and the These well-known benchmark SR methods include FRSCNN fissure in the fourth row in the figure cannot be clearly shown [12], LapSRN [16], ESPCN [15], and VDSR [13]. for the 3 CNN method. These fissure and middle peak features Encouragingly, our method outperformed these methods as are evidently shown in the A+ GAN predictions (shown in Fig. evidenced by the resultant SR images with higher PSNR and 5(b)), which are not used for the 3 CNN method. Only when all SSIM values. More importantly, our results faithfully present priors are combined in the ensemble learning, we can obtain more details and less noise in HR images. For example, in the results with good texture details. region pointed by the red arrows in the first two rows of Fig. 8, Since introduced in 2014 [42], GAN has been extensively texture details are clearer and much closer to the ground truth used in many applications due to its compelling performance while these textures are blurry in the images produced using the originated from the adversarial training mechanism. For SR other methods. Also, our results greatly reduced artifacts and MRI, despite substantial improvements the GAN model tends distortions. As pointed by the red arrows in the bottom two rows to induce artifacts and distortions in prediction images, which of Fig. 8, the shape of holes and fissure in our images were may due to the nature of GAN and the prevalence of closer to the ground truth than the competitive counterparts. background noise. On one hand, during the training process of Fig. 9 compares the pixel intensity accuracies between our a GAN model, the generator is trained to learn the true result and competing results. Here, we define the accuracy as distribution of training data. Compared with CNN-based the percentage of pixels with intensity close to the supervised training, the training process of GAN is not strongly corresponding pixels in HR images. supervised as predictions are controlled indirectly through a discriminator, not directly anchored with labels. Owing to this 𝑛𝑢𝑚 𝑐𝑢𝑟𝑦𝑎𝑐𝑎𝑐 = (10) mechanism, it is not easy to train the model so that it can 𝑛𝑢𝑚 𝑡𝑜𝑡𝑙𝑎 perfectly learn the true distribution. As a result, the generator can hardly produce a fake result that is exactly the same as a where 𝑛𝑢𝑚 is the number of accurate pixels and 𝑛𝑢𝑚 𝑡𝑜𝑡𝑙𝑎 real result. As shown in Fig. 3, the GAN suffer from edge is the number of all pixels, and the x-axis is the intensity distortions and ring-like or streak-like artifacts. On the other difference threshold. When the intensity difference between a hand, during the MR scanning process, noise is inevitably deblurred pixel and its corresponding pixel in the HR image is recorded [43], by which clinical MR images are greatly within this threshold, this pixel is counted as an accurate pixel. influenced. Noise hampers the training process of GAN so that For easy comparison, we quantized the all intensity values into it cannot be perfectly trained. This phenomenon is very evident the range from 0 to 255. Similar to the receiver operating when noise is prevalent that features cannot be easily extracted. characteristic (ROC) curve, it is clear in Fig. 9 that the closer a In Fig. 6(b), the subtle textures in the middle region are greatly curve is to the left top corner, the more desirable this curve. influenced by noise. As a result, there are substantial artifacts Compared to the other methods, the curve associated with our in each of the GAN SR predictions. Fortunately, these issues proposed ensemble learning method is the closest to the left top can be addressed by ensemble learning. By combining multiple corner, indicating that our method produces the results with GAN SR predictions, the size of the dataset was enlarged. most pixels close to the ground truth. When the threshold is 5, Through a CNN-based integrator, the robustness of the whole our method achieved an accuracy of over 90%, which is SR framework can be strengthened, effectively suppressing significantly higher than any one of the other methods. artifacts and distortions produced in each GAN SR prediction. Ensemble learning synergistically combines all GAN SR IV. DISCUSSIONS predictions together. Different from the simple average Based on the results described above, it can be seen that algorithm, the CNN-based integrator learns to perform through the ensemble learning superior MRI SR images can be ensemble learning in a data-driven fashion and is task-specific. achieved. In comparison with other state-of-the-art methods, Roughly speaking, the CNN can be assumed as a black box. our method produced impressive results without significant Through the CNN model, pixels in each input image is assigned artifacts, distortions, and blurry details, giving the highest for a weight for summation and nonlinear activation. During the PSNR and SSIM values. training process, the weight of each pixel in each type of GAN SR prediction can be learned according to high-level features 𝑎𝑐𝑐𝑢 𝑎𝑐𝑐𝑢 > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 9 [5] R. Hardie. "A fast image super-resolution algorithm using an adaptive and contexts. This data-driven and context-based mechanism Wiener filter." IEEE transactions on image processing, 2007, pp.2953- guarantees that the CNN-based ensemble learning method outperforms the commonly used averaging method. [6] J. V. Manjón, et al. "Non-local MRI upsampling." Medical image analysis, Since CNN can approximate a wide class of functions, it is 2010, pp.784-792. [7] J. Yang, et al. "Image super-resolution via sparse representation." IEEE understandable that artifacts and distortions shown in GAN SR transactions on image processing, 2010, pp.2861-2873. predictions can be greatly reduced through CNN-based [8] R. Zeyde, et al. "On single image scale-up using sparse-representations." ensemble learning. As discussed already, ensemble learning Proceedings of the international conference on curves and surfaces, 2010, enlarges the amount of data with diverse features and increase pp.711-730. [9] R. Timofte, et al. "A+: Adjusted anchored neighborhood regression for fast robustness accordingly. On the other hand, because most super-resolution." Proceedings of the Asian conference on computer vision, artifacts and distortions in each GAN SR prediction were 2014, pp.111-126. derived from processed images that were obtained using [10] C. Dong, et al. "Learning a deep convolutional network for image super- different approaches, these artifacts and distortions were resolution." Proceedings of the European conference on computer vision, 2014, pp.184-199. unlikely be the same or very coherent. Through ensemble [11] C. Dong, et al. "Image super-resolution using deep convolutional learning, these artifacts and distortions will be either effectively networks." IEEE transactions on pattern analysis and machine intelligence, canceled out or greatly reduced. This ability of alleviating 2015, pp.295-307. artifacts and distortions is a main advantage of our method [12] C. Dong, et al. "Accelerating the super-resolution convolutional neural network." Proceedings of the European conference on computer vision, compared with other state-of-the-art SR methods. 2016, pp.391-407. Another clear advantage of our proposed method is that our [13] J. Kim, et al. "Accurate image super-resolution using very deep results preserve details well. Assuming all subtle features are in convolutional networks." Proceedings of the IEEE conference on computer HR images, and these features are blurred or weakened in LR vision and pattern recognition, 2016, pp.1646-1654. [14] J. Kim, et al. "Deeply-recursive convolutional network for image super- images. After the processing step, in each processed image, resolution." Proceedings of the IEEE conference on computer vision and there are some features strengthened and other features further pattern recognition, 2016, pp.1637-1645. weakened. For example, in Case 1 in Fig. 3, edges were [15] W. Shi, et al. "Real-time single image and video super-resolution using an strengthened through ZIP, and texture was strengthened by A+. efficient sub-pixel convolutional neural network." Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.1874- Once most features are strengthened by at least one of the processing algorithms, it is in principle possible to collect a [16] W. Lai, et al. "Fast and accurate image super-resolution with deep relatively complete feature set and recover all details through laplacian pyramid networks." IEEE transactions on pattern analysis and ensemble learning. As a result, the final SR images after machine intelligence, 2018. [17] C. Ledig, et al. "Photo-realistic single image super-resolution using a ensemble learning are with clear details, outperforming generative adversarial network." Proceedings of the IEEE conference on individual model-based methods. computer vision and pattern recognition, 2017, pp.4681-4690. Compared to other model-based methods, our method is [18] X. Wang, et al. "ESRGAN: Enhanced super-resolution generative featured by ensemble learning that combines multiple neural adversarial networks." Proceedings of the European conference on computer vision, 2018. network models together, but it demands a higher [19] Y. Yuan, et al. "Unsupervised image super-resolution using cycle-in-cycle computational cost. generative adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.701-710. V. CONCLUSION [20] C. You, et al. "CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE)." arXiv preprint We have proposed an ensemble learning based SR method to 2018, arXiv:1808.04256. improve the image quality of MR images. We have first used 5 [21] B. Lim, et al. "Enhanced deep residual networks for single image super- resolution." Proceedings of the IEEE conference on computer vision and existing SR algorithms to enlarge LR images and obtained PLR pattern recognition, 2017, pp.136-144. images with complementary priors. Then, a GAN framework [22] T. Tong, et al. "Image super-resolution using dense skip connections." has been used to produce SR prediction results based on Proceedings of the IEEE international conference on computer vision, different PLR images. Next, we can obtain superior SR images 2017, pp.4799-4807. [23] Y. Zhang, et al. "Residual dense network for image super-resolution." through a CNN-based ensemble learning. The SR images Proceedings of the IEEE conference on computer vision and pattern obtained after ensemble learning is better than any individual recognition, 2018, pp.2472-2481. PLR images or GAN SR prediction results. Compared with [24] Y. Zhang, et al. "Image super-resolution using very deep residual channel other state-of-the-art methods, our method enjoys advantages attention networks." Proceedings of the European conference on computer vision, 2018, pp.286-301. of minimal artifacts, little distortions, insignificant noise, and [25] W. Han, et al. "Image super-resolution via dual-state recurrent networks." rich details. Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.1654-1663. REFERENCES [26] M. S. Sajjadi, et al. "Frame-recurrent video super-resolution." Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, [1] E. Plenge, et al. "Super‐resolution methods in MRI: can they improve the pp.6626-6634. trade‐off between resolution, signal‐to‐noise ratio, and acquisition time?" [27] Y. Huang, et al. "Bidirectional recurrent convolutional networks for multi- Magnetic resonance in medicine, 2012, pp.1983-1993. frame super-resolution." Advances in neural information processing [2] E. V. Reeth, et al. "Super‐resolution in magnetic resonance imaging: a systems, 2015, pp.235-243. review." Concepts in magnetic resonance part a, 2012, pp.306-325. [28] A. S. Chaudhari, et al. "Super‐resolution musculoskeletal MRI using deep [3] S. C. Park, et al. "Super-resolution image reconstruction: a technical learning." Magnetic resonance in medicine, 2018, pp.2139-2154. overview." IEEE signal processing magazine, 2003, pp.21-36. [29] K. Zeng, et al. "Simultaneous single-and multi-contrast super-resolution [4] X. Li, et al. "New edge-directed interpolation." IEEE transactions on image for brain MRI images based on a convolutional neural network." processing, 2001, pp.1521-1527. Computers in biology and medicine, 2018, pp.133-141. [30] Y. Chen, et al. "Efficient and accurate MRI super-resolution using a generative adversarial network and 3D multi-Level densely connected > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 10 network." Proceedings of the international conference on medical image over 100 papers to theory, algorithms, artifact reduction and computing and computer-assisted intervention, 2018, pp.91-99. biomedical applications in this important area of CT research. [31] Y. Chen, et al. "Brain MRI super resolution using 3D deep densely Currently, there are 100+ million medical CT scans yearly with connected neural networks." Proceedings of the 2018 IEEE 15th international symposium on biomedical imaging, 2018, pp.739-742. a majority in the spiral/helical cone-beam/multi-slice mode. Dr. [32] C. Liu, et al. "Fusing multi-scale information in convolution network for Wang’s group developed interior tomography theory and MR image super-resolution reconstruction." Biomedical engineering algorithms to solve the long-standing “interior problem” for online, 2018, pp.114. high-fidelity local reconstruction, enabling omni-tomography [33] C. Zhang, et al. "Ensemble machine learning: methods and applications." Springer Science & Business Media, 2012. (“all-in-one”) with CT-MRI as an example. He initiated the area [34] T. G. Dietterich. "Ensemble methods in machine learning." International of bioluminescence tomography with a significant impact on workshop on multiple classifier systems, 2000, pp.1-15. biophotonics. He has published 460+ journal publications, [35] L. Wang, et al. "Ensemble based deep networks for image super- covering imaging-related diverse topics at depth and receiving resolution." Pattern recognition, 2017, pp.191-198. [36] R. Liao, et al. "Video super-resolution via deep draft-ensemble learning." a high number of citations and academic awards. His results Proceedings of the IEEE international conference on computer vision, were featured in Nature, Science, PNAS, and various news 2015, pp.531-539. media. In 2016, he published the first perspective on deep- [37] M. A. Bernstein, et al. "Effect of windowing and zero‐filled reconstruction learning-based tomographic imaging as the new direction of of MRI data on spatial resolution and acquisition strategy." Journal of magnetic resonance imaging: an official journal of the international machine learning and the Nature Machine Intelligence paper on society for magnetic resonance in medicine, 2001, pp.270-280. the superiority of deep learning over iterative reconstruction. [38] I. Gulrajani, et al. "Improved training of Wasserstein GANs." Advances in His team has been continuously well-funded by federal neural information processing systems, 2017, pp.5767-5777. agencies and major imaging companies, actively translating [39] H. Shan, et al. "3-D convolutional encoder-decoder network for low-dose CT via transfer learning from a 2-D trained network." IEEE transactions deep learning techniques into imaging products. His research on medical imaging, 2018, pp.1522-1534. interests include x-ray CT, MRI, optical tomography, [40] Q. Yang, et al. "Low-dose CT image denoising using a generative multimodality fusion, and machine learning. He is the Lead adversarial network with Wasserstein distance and perceptual loss." IEEE Guest Editor for five IEEE Transactions on Medical Imaging transactions on medical imaging, 2018, pp.1348-1357. [41] J. Zbontar, et al. "fastMRI: An open dataset and benchmarks for Special Issues, Founding Editor-in-Chief of International accelerated MRI." arXiv preprint, 2018, arXiv:1811.08839. Journal of Biomedical Imaging, Board Member of IEEE [42] I. Goodfellow, et al. "Generative adversarial nets." Advances in neural Access, and Associate Editor of IEEE Trans. Medical Imaging information processing systems, 2014, pp.2672-2680. (TMI) (recognized as “Outstanding Associate Editor” by TMI), [43] H. Gudbjartsson, et al. "The Rician distribution of noisy MRI data." Magnetic resonance in medicine, 1995, pp.910-914. Medical Physics, and Machine Learning Science and [44] M. Lin, et al. "Network in network." arXiv preprint, 2013, Technology. He is Fellow of IEEE, SPIE, OSA, AIMBE, arXiv:1312.4400. AAPM, and AAAS. Qing Lyu received his Bachelor of Science degree and Master of Science degree in the department of biomedical engineering in Shanghai Jiao Tong University in 2013 and 2016 respectively. In 2017, he joined Dr. Ge Wang’s lab. His main research interests include the MRI postprocessing and signal optimization. Hongming Shan received the B.S. degree from Shandong University of Technology, China, in 2011, and the Ph.D. degree from Fudan University, China, in 2017. He is currently a Postdoctoral Scholar with the Rensselaer Polytechnic Institute, Troy, New York, USA. His research interests include machine/deep learning, computer vision, dimension reduction, and biomedical imaging. Ge Wang is the Clark & Crossan Endowed Chair Professor and Director of the Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY, USA. He published the first spiral/helical cone- beam/multi-slice CT algorithm in 1991 and has since then systematically contributed http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Electrical Engineering and Systems Science arXiv (Cornell University)

MRI Super-Resolution with Ensemble Learning and Complementary Priors

Download PDF
10 pages

Loading next page...
 
/lp/arxiv-cornell-university/mri-super-resolution-with-ensemble-learning-and-complementary-priors-kyJ1larflE

References (48)

ISSN
2333-9403
eISSN
ARCH-3348
DOI
10.1109/TCI.2020.2964201
Publisher site
See Article on Publisher Site

Abstract

> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 1 MRI Super-Resolution with Ensemble Learning and Complementary Priors Qing Lyu, Hongming Shan, and Ge Wang, Fellow, IEEE protocol [1], [2]. Abstract—Magnetic resonance imaging (MRI) is a widely used SR has been a hot topic in the computational vision field over medical imaging modality. However, due to the limitations in the past decades. A large number of methods were proposed to hardware, scan time, and throughput, it is often clinically improve image resolution retrospectively. Roughly speaking, challenging to obtain high-quality MR images. The super- super-resolution algorithms can be divided into the two resolution approach is potentially promising to improve MR image categories: model-based and learning-based. Interpolation quality without any hardware upgrade. In this paper, we propose algorithms [3], [4], like bilinear, bicubic, and nearest neighbor an ensemble learning and deep learning framework for MR image super-resolution. In our study, we first enlarged low resolution interpolation techniques, are representatives of simple model- images using 5 commonly used super-resolution algorithms and based approaches, which can be directly used to enlarge images. obtained differentially enlarged image datasets with Wiener filtering and iterative deblurring algorithms utilize complementary priors. Then, a generative adversarial network knowledge on the system point spread function (PSF) to recover (GAN) is trained with each dataset to generate super-resolution image resolution [5], [6], and are also considered as model- MR images. Finally, a convolutional neural network is used for based algorithms. Dictionary learning-based super-resolution ensemble learning that synergizes the outputs of GANs into the final MR super-resolution images. According to our results, the techniques [7]-[9] are examples of learning-based algorithms. ensemble learning results outcome any one of GAN outputs. Recently, with the rapid evolution of machine learning, Compared with some state-of-the-art deep learning-based super- especially deep learning, deep neural networks have become resolution methods, our approach is advantageous in suppressing popular among SR studies. In this context, diverse neural artifacts and keeping more image details. network architectures were designed and tested. Successful architectures include convolutional neural networks (CNNs) Index Terms— Deep learning, ensemble learning, generative adversarial network (GAN), magnetic resonance imaging (MRI), [10]-[16], generative adversarial networks (GANs) [17]-[20], super-resolution residual neural networks (ResNets) [21]-[24], and recurrent neural networks (RNNs) [25]-[27]. Specifically, for MRI SR studies, models based on CNN, ResNet, and GAN were I. INTRODUCTION reported [28]-[32]. Although there were a plenty of decent SR results in the literature, bothersome artifacts or distortions were agnetic resonance imaging (MRI) is one of the most M often observed in the SR results produced by these methods. commonly used medical imaging modalities over the Therefore, the challenge remains for achieving superior SR on world. It noninvasively depicts structural and functional MR images. In other words, MRI SR research must deliver features inside a patient in rich contrasts. Compared with other images with faithful details and little artifacts in order to be medical imaging modalities such as computed tomography clinically relevant. Since the existing methods cannot perfectly (CT) and nuclear imaging, MRI does not involve ionizing satisfy such a requirement, it is necessary to develop new SR radiation. Furthermore, MRI plays a dominating role in methods based on all encouraging results. neurological imaging especially brain research. Among important deep learning strategies, ensemble deep However, MRI suffers from major physical limitations in learning is a meta-method for further enhancement of deep background magnetic field, gradient fields, imaging speed, and learning results [33], [34]. By combining multiple models, signal-to-noise ratio. Hence, the pixel size of a clinical MR ensemble learning is expected to produce synergistic results image is often constrained to an order of millimeters. To obtain than the results from individual models. Merits of ensemble higher resolution MR images, either a more complicated MRI learning were successfully demonstrated to improve image system with stronger background magnetic field or a longer quality [35], [36]. In this MRI SR study, by combining multiple scan time needs to be used, which will dramatically increase GAN models trained on different image prior datasets, we expenses and is rarely clinically applicable. To address this demonstrate better MRI SR image quality than that associated problem, the super-resolution (SR) approach holds a great with any single model. In this paper, we describe a promise that needs to change neither hardware nor scanning (Corresponding author: Ge Wang) All authors are with the Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, 12180, USA (email: [email protected], [email protected], [email protected]) > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 2 Fig. 1. Overall workflow of the SR method proposed in this paper. convolutional neural network to integrate diverse SR results datasets, each of which corresponds to one of the five SR from individual models. Our method produces better image algorithms. These image datasets are separately fed into the quality than state-of-the-art SR models, being capable of corresponding GAN for training and then to produce five SR detailing textures and avoiding artifacts. counterparts. The final step is to use a CNN to do the ensemble learning and obtain the final MRI SR result. II. METHODOLOGY A. Down-sampling Differing from commonly used down-sampling strategy that 𝑝 ×𝑞 Assuming that 𝐼 ∈ ℝ is a low-resolution (LR) MR 𝐿𝑅 down samples data in the image domain, we chose to down- 𝑚 ×𝑛 image of the size 𝑝 × 𝑞 and 𝐼 ∈ ℝ the corresponding 𝐻𝑅 sample data in the frequency domain. The original HR images high-resolution (HR) MR image of the size 𝑚 × 𝑛 . The were converted into the frequency domain via the 2D Fourier relationship between them can be expressed as transform. Only the central 25% of the data points were kept in the k-space, while all peripheral data points were zeroed out. 𝐼 = 𝑓 (𝐼 ) (1) Then, the cropped dataset was converted back into an image 𝐿𝑅 𝐻𝑅 through the 2D inverse Fourier transform as a LR image. Using 𝑚 ×𝑛 𝑝 ×𝑞 this down-sampling method, we obtained LR images with 2- where 𝑓 : 𝐼 ∈ ℝ → 𝐼 ∈ ℝ denotes the down- 𝐿𝑅 fold resolution degradation, which means 𝑝 = 𝑚 /2 and 𝑞 = sampling/blurring process that creates a LR counterpart from a 𝑛 /2. HR image. The overall SR process is to implement an −1 approximate inverse function 𝑔 ≈ 𝑓 B. Processing Before LR images are fed into the neural network, we adopt −1 𝑔 ≈ 𝑓 : 𝐼 ≈ 𝑔 (𝐼 ) (2) 𝐻𝑅 𝐿𝑅 existing SR algorithms to enlarge a LR image to the size of the target HR image. This processing step was also used in some As an ill-posed problem, it is impossible to find this inverse SR studies like VDSR [13] and SRCNN [11]. With such a function exactly. Instead, priors can be used to acquire an processing step, the quality of enlarged images can be somehow −1 approximation of 𝑓 . Multiple types of priors are used in this improved so that it is easier for neural networks to extract image −1 study to approximate 𝑓 as closely to the ideal function as features and obtain better prediction results. Moreover, we can possible. establish datasets with complementary priors through using Fig. 1 presents the proposed overall workflow of SR method. multiple SR algorithms in this processing step. In this paper, in In our scheme, we first down-sample original HR images and order to distinguish enlarged images with the latter neural obtain LR counterparts, and then use five different model-based network results, we call enlarged images “processed LR images SR algorithms to process LR images and obtain five image (PLR)” rather than SR images. In this work, 5 classic SR 𝐻𝑅 > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 3 Fig. 2. The architecture of the proposed GAN. (a) The generator and (b) the discriminator. algorithms were used to enlarge LR images and obtain the and HR images, adding such skip connections can greatly boost corresponding PLRs, which are zero interpolation filling (ZIP) the training process and keep the generated images structurally [37], bicubic interpolation (BI) [3], new edge directed similar to the input LR images. Similarly, the discriminator interpolation (NEDI) [4], sparse coding (SC) [7], and adjusted consists of five blocks followed by a global average pooling anchored neighborhood regression (A+) [9]. ZIP is a simplest (GAP) layer. Each block includes convolution, layer super-resolution method that adds zeros in the peripheral region normalization, and leaky-ReLU (LReLU). Note that we replace in the frequency domain. It is widely used to enlarge MR the fully-connected layer with a GAP layer to reduce the images. BI and NEDI are two improved interpolation-based amount of parameters in the neural network and facilitate the algorithms, while SC and A+ are two dictionary-learning-based training process [44]. Each block in the discriminator performs algorithms. In the A+ algorithm, 1,024 dictionary atoms were convolution, layer normalization and ReLU activation function used, which were obtained from a training dataset of 0.5 million similar to what the block in the generator does. The details of natural images. Similarly, the SC method also used a dictionary the generator and the discriminator are listed in Table I and II. contained 1,024 atoms. Using these different SR algorithms, The objective function of the generator is as follows: different enlarged images were created, each with its unique priors. Next, we used these enlarged images as the inputs to (4) min ℒ = ℒ + 𝜆 ℒ + 𝜆 ℒ + 𝜆 ℒ 𝐺 𝑑𝑣𝑎 1 𝑟𝑎𝑔 2 𝑠𝑒𝑚 3 𝑝𝑟𝑒 train the corresponding GAN models separately. Each ℒ = −𝔼 [𝐷 (𝐺 (𝐼 ))] (5) 𝑑𝑣𝑎 𝐼 𝑃𝐿𝑅 enlarging/deblurring process can be expressed as 𝑅𝐿𝑃 ℒ = 𝔼 ‖∇(𝐺 (𝐼 )) − ∇(𝐼 )‖ (6) 𝑟𝑎𝑔 (𝐼 ,𝐼 ) 𝑃𝐿𝑅 𝐻𝑅 𝑅𝐿𝑃 𝐻𝑅 𝐼 = ℎ (𝐼 ) (3) 𝑃𝐿𝑅 ,𝑖 𝑖 𝐿𝑅 ℒ = 𝔼 ‖∅(𝐺 (𝐼 )) − ∅(𝐼 )‖ (7) 𝑝𝑟𝑒 (𝐼 ,𝐼 ) 𝑃𝐿𝑅 𝐻𝑅 𝑅𝐿𝑃 𝐻𝑅 𝑚 ×𝑛 where 𝐼 ∈ ℝ is a deblurred/enlarged image. ℎ : 𝐼 ∈ 𝑃𝐿𝑅 ,𝑖 𝑖 𝐿𝑅 𝑝 ×𝑞 𝑚 ×𝑛 ℝ → 𝐼 ∈ ℝ denotes the processing process. As 𝑃𝐿𝑅 ,𝑖 where 𝜃 stands for the generator parameters. ℒ is the 𝐺 𝑑𝑣𝑎 there were 5 SR algorithms used in this processing step, 𝑖 = adversarial loss. ℒ is the gradient loss, which is the L2 norm 𝑟𝑎𝑔 1,2, … ,5. difference between gradients of the generated image and its C. GAN model ground truth. ∇ stands for the gradient calculated in both x and y axis. ℒ and ℒ are the mean squared error (MSE) and The GAN model with Wasserstein distance and gradient 𝑠𝑒𝑚 𝑝𝑟𝑒 penalty (WGAN-GP) [38] is selected in this study. The the perceptual loss respectively. MSE reflects the image content generator used in this study is a modified version of the similarity between generated output images and processed input encoder-decoder that was originally used for CT denoising [39]. images at the pixel level, while the perceptual loss shows the The architectures of the generator and the discriminator are similarity in a high-level feature space. We use the pretrained shown in Fig. 2. For the generator, it contains seven blocks with VGG19 model to calculate the perceptual loss [39], [40], and ∅ th three skip connections. Each block involves convolution, layer represents the feature map of the 16 convolutional layer in the normalization (LN), and ReLU activation operations VGG19 model. As the mean-squared-error loss tends to blur sequentially. As mentioned in [38], batch normalization cannot images, the gradient loss is introduced to penalize any blurring be used in the WGAP-GP model, instead, we utilize layer and enhance the edge. The objective function of the normalization. With skip connections, feature maps in the discriminator is the standard WGAN-GP equation shown in previous blocks are concatenated to the feature maps in the [38]. Overall, this GAN SR process is then expressed as latter blocks. Because of the structural similarity between LR > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 4 𝐼 = 𝐺 (𝐼 ) = 𝜑 (𝐼 , 𝜃 ) (8) A. Experimental implementation and neural network ,𝑖 𝑃𝐿𝑅 ,𝑖 𝑃𝐿𝑅 ,𝑖 𝐺 ,𝑖 convergence 𝑚 ×𝑛 where 𝐼 ∈ ℝ is a output image from generator in GAN, The NYU fastMRI dataset was used in this study [41]. All ,𝑖 𝑚 ×𝑛 𝑚 ×𝑛 images were reconstructed from proton density weighted knee 𝜑 : 𝐼 ∈ ℝ → 𝐼 ∈ ℝ denotes a GAN deblurring 𝑃𝐿𝑅 ,𝑖 ,𝑖 scans under 1.5 or 3 Tesla. The size of original HR images is process for an SR image, 𝜃 represents the parameters of a 𝐺 ,𝑖 320 × 320. Totally, 159 patient scans with 5,744 slices were GAN model. As there are 5 SR algorithms used in this step, 𝑖 = used for training (80%) and validation (20%), and additional 43 1,2, … ,5. patient scans with 1,500 slices for testing. Before training, each processed image was decomposed into forty-nine 80 × 80 TABLE I STRUCTURAL DETAILS OF THE GENERATOR, WHERE n1s1 MEANS patches. During the training process of the network models, 1 CONVOLUTION KERNELS WITH STRIDE OF 1. weights in the objective function were selected to be 0.1, 0.1, Block1 Block2 Block3 Block4 Block5 Block6 Block7 and 1 for 𝜆 , 𝜆 , and 𝜆 respectively via the grid searching 1 2 3 n32s1 n64s1 n128s1 n256s1 n128s1 n64s1 n32s1 strategy. The training continued for 50 epochs with the learning n32s1 n64s1 n128s1 n256s1 n128s1 n64s1 n1s1 -5 rate of 2×10 and batch size of 64 for each GAN model. During the training process, to visualize the convergence of the TABLE II STRUCTURAL DETAILS OF THE DISCRIMINATOR, WHERE n1s1 involved models, the values of MSE and perceptual loss are MEANS 1 CONVOLUTION KERNELS WITH STRIDE OF 1. recorded and showed in Fig. 3. For ensemble learning, the CNN Block1 Block2 Block3 Block4 Block5 -5 was trained over 80 epochs with the learning rate of 2×10 and n64s2 n128s2 n256s2 n512s1 n1s1 batch size of 4. The experiment was conducted in Tensorflow on a single GTX 1080Ti GPU. According to the MSE and perceptual loss curves in Fig. 3, D. Ensemble learning all the models converge well during the training process since Ensemble learning is to synergize diverse information all the curves become gradually steady with the number of sources for the best results. In this study, SR images training epochs. Compared with the MSE curves, it can be individually obtained can be integrated via ensemble learning found that the curve of A+ is lower than the other four curves, to improve SR MR image quality. Specifically, the final SR which shows that the A+ based GAN model can achieve the results are derived from different types of GAN SR predictions. best SR image with the highest similarity at the pixel level. In In the processing step, we utilized the five SR algorithms to the high-level space, as extracted by the perceptual loss, both process LR images and obtain five sets of enlarged/deblurred the A+ and ZIP curves show the lowest values while the NEDI images. Then, we train five GAN models (with the same curve is the highest. However, all the curves are pretty close to structure) separately corresponding to each of these five each other after 40 epochs. algorithms and then obtain five sets of prediction results. After that, a CNN model is used to integrate all these images through ensemble learning. The structure of the CNN is the same as the generator in the proposed GAN. Images from each type of prediction results are concatenated before input to the CNN for training, validation, and testing. The mean-absolute-error (MAE) is used in the loss function during the training process. The ensemble learning process can be expressed as 𝐼 = 𝜙 (∑ 𝐼 , Θ) (9) ,𝑖 Fig. 3. Comparisons between the involved model types during the training process in terms of (a) MSE and (b) perceptual loss. 𝑚 ×𝑛 where 𝐼 ∈ ℝ denotes ensemble learning results which are 𝑚 ×𝑛 𝑚 ×𝑛 also our SR outcomes, 𝜙 : 𝐼 ∈ ℝ → 𝐼 ∈ ℝ denotes ,𝑖 B. Processed results the ensemble learning process, Θ represents the parameters of It is shown in Fig. 4-6 that PLR images had smoother edges and the CNN model. clearer shapes compared with the corresponding LR images. As different SR algorithms were used in the processing step, PLR III. RESULTS images obtained from these SR algorithms were somehow Our ensemble learning SR approach was streamlined and different. As far as the edges in Fig. 5(a) are concerned, it can evaluated for MRI SR. It produced SR images with more be found that all the PLR images give clearer edges than the structural details, less artifacts and distortion. original LR image. Among all the 5 processed images, ZIP and In the following, we will show our processed images, GAN A+ images are closer to the HR image than the other three predictions, and ensemble learning results, and compare our SR image types. However, these PLR images also introduced some results with state-of-the-art methods. To evaluate image quality, artifacts and distortions. Some edges in the NEDI PLR image the metrics we used include structural similarity (SSIM) and were slightly distorted, with ring or streak artifacts in the ZIP peak signal-to-noise ratio (PSNR). 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 𝑆𝑅 > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 5 Fig. 4. Input, intermediate and output images in the SR workflow in (a) Cases 1 and (b) 2, where processed results are in the yellow boxes, LR images are in the purple boxes, HR images in the green boxes, ensemble learning results in the red boxes, and GAN predictions in the orange boxes. For the processed results and GAN predictions from left to right are ZIP, BI, NEDI, SC, A+. White dotted boxes are zoomed in region shown in Fig. 5 and 6. Fig. 5. Zoomed in results of regions marked by white dotted boxes in Cases 1, where processed results are in the yellow boxes, LR images are in the purple boxes, HR images in the green boxes, ensemble learning results in the red boxes, and GAN predictions in the orange boxes. For the processed results and GAN predictions from left to right are ZIP, BI, NEDI, SC, A+. > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 6 Fig. 6. Zoomed in results of regions marked by white dotted boxes in Cases 2, where processed results are in the yellow boxes, LR images are in the purple boxes, HR images in the green boxes, ensemble learning results in the red boxes, and GAN predictions in the orange boxes. For the processed results and GAN predictions from left to right are ZIP, BI, NEDI, SC, A+. TABLE III QUANTITATIVE EVALUATION OF THE PROCESSED IMAGES, GAN PREDICTIONS AND ENSEMBLE LEARNING RESULTS IN FIG. 5 AND 6. Preprocessed results GAN predictions Ensemble ZIP BI NEDI SC A+ ZIP BI NEDI SC A+ learning Figure SSIM 0.86 0.81 0.84 0.80 0.86 0.88 0.88 0.87 0.87 0.87 0.95 4(a) PSNR 25.82 25.92 23.53 24.22 24.52 26.67 28.10 26.07 27.71 27.41 27.61 SSIM 0.86 0.82 0.84 0.80 0.86 0.88 0.88 0.86 0.87 0.87 0.95 Figure 4(b) PSNR 28.05 26.02 26.69 25.86 28.36 28.52 28.03 26.79 28.04 27.43 32.67 Figure SSIM 0.85 0.79 0.82 0.77 0.85 0.86 0.86 0.85 0.86 0.86 0.92 5(a) PSNR 25.23 23.49 24.96 23.92 25.95 26.90 26.73 25.74 26.34 27.15 29.44 Figure SSIM 0.71 0.62 0.66 0.61 0.71 0.70 0.71 0.68 0.69 0.71 0.82 5(b) PSNR 24.20 22.57 23.65 21.87 24.28 24.25 24.55 24.10 24.22 24.37 25.98 and SC PLR images. In Fig. 5(b), for each PLR image the are also presented in the GAN SR predictions. As the GANs texture details became smoother while holes in the peripheral were trained on different training datasets, when GAN SR region were more distinguishable compared with the LR image. predictions are compared, differences among them are clearly Despite some artifacts introduced, the SC image contains the detectable for the same ground truth. For example, when clearest hole shapes. Each algorithm had its own strengths and comparing edges pointed by the red arrow in Fig. 5(a), it can be weaknesses. After the processing step, we obtained diverse found that shapes of edges in the GAN SR predictions were enlarged/deblurred images, some of which present edges well slightly different. Another example was shown in Fig. 6(a), while the others of which preserve details satisfactorily. shapes of the hole pointed by the red arrows were moderately different from each other. These shape deformations can be C. Super-resolution results through GAN called distortions. The quality of the prediction images was After training the GAN models separately, five types of GAN quantified in Table III. It can be seen in the table that despite SR prediction results were obtained, as shown in each purple the image quality seems visually improved in those SR box in in Fig. 3. Compared with the PLR results, the image prediction results, the PSNR and SSIM values of the GAN quality of GAN SR predictions was improved. Generally, they predictions were just close to or slightly higher than the PLR included more details, clearer edges, and less blurs, which are results. The appearance of artifacts and distortions in the SR much closer to the original HL images than the PLR versions. predictions contributed to compromise the PSNR and SSIM For GAN SR predictions, edges in Fig. 5(a) become much values. After training, the GAN SR predictions produced clearer while textures in Fig. 6(b) are less blurred compared to numerous tiny artifacts and distortions. For example, the SSIM the corresponding edges and textures in PLR images. However, value of the ZIP GAN prediction was even lower than the just as shown in the PLR images, some artifacts and distortions > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 7 corresponding PLR image. When the two images are compared, used for the ensemble learning, denoted as 3 CNN. Moreover, despite textures in the GAN SR prediction image were clearer simple averaging algorithm was used to replace the CNN and closer to the HR image, there were streak artifacts model. According to the amount of GAN SR predictions used, produced. As a result, the SSIM value was decreased after they are referred as 3 averaging and 5 averaging. Representative training. results are presented in Fig. 7 and Table IV. It can be found that the combination of 3 types of GAN SR predictions yielded less D. Super-resolution results through ensemble learning PSNR and SSIM values than the integration of 5 types of GAN Compared with GAN SR predictions, the most distinct SR predictions. Importantly, there were more artifacts characteristic of the ensemble learning results is that artifacts noticeable in the combination of the 3 types of ensemble and noise were greatly reduced as shown in Fig. 4-6. Also, the learning results, as pointed by the red arrows in Fig. 7. All these ensemble learning results showed richer textural details than results demonstrate that ensemble learning results can be any GAN SR predictions, for example, when comparing the further improved by combining more GAN SR predictions. regions pointed by the red arrows in Fig. 5(b), it can be found When comparing the result from simple averaging followed by that the middle peak can be well presented after ensemble CNN-based integration, the use of CNN for ensemble learning learning. In addition, ensemble learning essentially avoided edge distortions, such as the edges pointed by the red arrows in Fig. 5(a) were very close to the ground truth. Quantitatively, the ensemble learning results achieved the highest PSNR and SSIM values among all the results, indicating that ensemble learning greatly improved the image quality. Fig. 8. Comparison of the proposed method and state-of-the-art methods. TABLE V QUANTITATIVE EVALUATION OF THE RESULTS IN FIG. 8. FSRCNN LapSRN ESPCN VDSR Ours SSIM 0.88 0.88 0.81 0.89 0.95 Case1 PSNR 25.90 25.68 21.14 26.96 27.61 SSIM 0.88 0.88 0.86 0.89 0.95 Case2 PSNR 28.33 28.63 24.51 28.25 32.67 SSIM 0.86 0.86 0.83 0.88 0.92 Case3 PSNR 25.58 25.22 21.57 26.12 29.44 Fig. 7. Ensemble learning results in Cases 1-4, where k averaging means that SSIM 0.70 0.71 0.66 0.71 0.82 the result is simple average of k GAN prediction results, and k CNN means that Case4 PSNR 24.04 23.94 21.06 24.11 25.98 ensemble learning is done through the CNN combining k prediction results. TABLE IV QUANTITATIVE EVALUATION OF THE RESULTS IN FIG. 7. 3 average 3 CNN 5 average 5 CNN SSIM 0.90 0.91 0.90 0.95 Case1 PSNR 26.20 28.24 26.45 27.61 SSIM 0.89 0.91 0.90 0.95 Case2 PSNR 28.38 28.90 28.72 32.67 SSIM 0.86 0.88 0.87 0.92 Case3 PSNR 25.86 26.23 25.97 29.44 SSIM 0.69 0.70 0.71 0.82 Case4 PSNR 23.75 24.20 23.95 25.98 To investigate how much improvement can be made by the ensemble learning, we further compared our results from ensemble learning with different methods. Instead combining 5 types of GAN SR predictions for ensemble learning, only three types of GAN SR predictions were used. Only GAN SR Fig. 9. Pixel intensity accuracy of the proposed method and state-of-the-art predictions trained from ZIP, BI and NEDI PLR images were methods. > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 8 can achieve better results, showing that optimal integration of In the processing step, 5 different SR algorithms were used diverse SR style would improve image quality further. to enlarge LR images. As a result, 5 types of enlarged PLR image datasets were obtained, each with its unique image priors. From Fig. 4-6, it can be found that each SR algorithm TABLE VI can somehow closely preserve some part of image features. For COMPARISON OF OUR METHOD AND OTHER STATE-OF-THE-ART example, ZIP, BC, and NEDI algorithms are good at presenting METHODS (MEAN±STD). edges with clear and smooth shapes, SC and A+ PLR images FSRCNN LapSRN ESPCN VDSR Ours show texture details. By combining all these SR algorithms SSIM 0.87±0.02 0.88±0.02 0.83±0.03 0.88±0.02 0.92±0.01 PSNR 28.87±2.33 28.80±2.38 24.93±3.30 28.99±2.25 31.35±2.15 together, we can create PLR datasets that are with complementary image priors and finally obtain good SR results that are very close to the ground truth after ensemble learning. Fig. 7 shows the necessity of combining all priors. When only E. Comparison with other deep learning-based methods ZIP and interpolation-based methods were used, ensemble Fig. 8-9 and Table V-VI compare our results with the learning results would not well present texture details. For counterparts obtained using other state-of-the-art SR methods. example, the middle peak in the second row in Fig. 7 and the These well-known benchmark SR methods include FRSCNN fissure in the fourth row in the figure cannot be clearly shown [12], LapSRN [16], ESPCN [15], and VDSR [13]. for the 3 CNN method. These fissure and middle peak features Encouragingly, our method outperformed these methods as are evidently shown in the A+ GAN predictions (shown in Fig. evidenced by the resultant SR images with higher PSNR and 5(b)), which are not used for the 3 CNN method. Only when all SSIM values. More importantly, our results faithfully present priors are combined in the ensemble learning, we can obtain more details and less noise in HR images. For example, in the results with good texture details. region pointed by the red arrows in the first two rows of Fig. 8, Since introduced in 2014 [42], GAN has been extensively texture details are clearer and much closer to the ground truth used in many applications due to its compelling performance while these textures are blurry in the images produced using the originated from the adversarial training mechanism. For SR other methods. Also, our results greatly reduced artifacts and MRI, despite substantial improvements the GAN model tends distortions. As pointed by the red arrows in the bottom two rows to induce artifacts and distortions in prediction images, which of Fig. 8, the shape of holes and fissure in our images were may due to the nature of GAN and the prevalence of closer to the ground truth than the competitive counterparts. background noise. On one hand, during the training process of Fig. 9 compares the pixel intensity accuracies between our a GAN model, the generator is trained to learn the true result and competing results. Here, we define the accuracy as distribution of training data. Compared with CNN-based the percentage of pixels with intensity close to the supervised training, the training process of GAN is not strongly corresponding pixels in HR images. supervised as predictions are controlled indirectly through a discriminator, not directly anchored with labels. Owing to this 𝑛𝑢𝑚 𝑐𝑢𝑟𝑦𝑎𝑐𝑎𝑐 = (10) mechanism, it is not easy to train the model so that it can 𝑛𝑢𝑚 𝑡𝑜𝑡𝑙𝑎 perfectly learn the true distribution. As a result, the generator can hardly produce a fake result that is exactly the same as a where 𝑛𝑢𝑚 is the number of accurate pixels and 𝑛𝑢𝑚 𝑡𝑜𝑡𝑙𝑎 real result. As shown in Fig. 3, the GAN suffer from edge is the number of all pixels, and the x-axis is the intensity distortions and ring-like or streak-like artifacts. On the other difference threshold. When the intensity difference between a hand, during the MR scanning process, noise is inevitably deblurred pixel and its corresponding pixel in the HR image is recorded [43], by which clinical MR images are greatly within this threshold, this pixel is counted as an accurate pixel. influenced. Noise hampers the training process of GAN so that For easy comparison, we quantized the all intensity values into it cannot be perfectly trained. This phenomenon is very evident the range from 0 to 255. Similar to the receiver operating when noise is prevalent that features cannot be easily extracted. characteristic (ROC) curve, it is clear in Fig. 9 that the closer a In Fig. 6(b), the subtle textures in the middle region are greatly curve is to the left top corner, the more desirable this curve. influenced by noise. As a result, there are substantial artifacts Compared to the other methods, the curve associated with our in each of the GAN SR predictions. Fortunately, these issues proposed ensemble learning method is the closest to the left top can be addressed by ensemble learning. By combining multiple corner, indicating that our method produces the results with GAN SR predictions, the size of the dataset was enlarged. most pixels close to the ground truth. When the threshold is 5, Through a CNN-based integrator, the robustness of the whole our method achieved an accuracy of over 90%, which is SR framework can be strengthened, effectively suppressing significantly higher than any one of the other methods. artifacts and distortions produced in each GAN SR prediction. Ensemble learning synergistically combines all GAN SR IV. DISCUSSIONS predictions together. Different from the simple average Based on the results described above, it can be seen that algorithm, the CNN-based integrator learns to perform through the ensemble learning superior MRI SR images can be ensemble learning in a data-driven fashion and is task-specific. achieved. In comparison with other state-of-the-art methods, Roughly speaking, the CNN can be assumed as a black box. our method produced impressive results without significant Through the CNN model, pixels in each input image is assigned artifacts, distortions, and blurry details, giving the highest for a weight for summation and nonlinear activation. During the PSNR and SSIM values. training process, the weight of each pixel in each type of GAN SR prediction can be learned according to high-level features 𝑎𝑐𝑐𝑢 𝑎𝑐𝑐𝑢 > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 9 [5] R. Hardie. "A fast image super-resolution algorithm using an adaptive and contexts. This data-driven and context-based mechanism Wiener filter." IEEE transactions on image processing, 2007, pp.2953- guarantees that the CNN-based ensemble learning method outperforms the commonly used averaging method. [6] J. V. Manjón, et al. "Non-local MRI upsampling." Medical image analysis, Since CNN can approximate a wide class of functions, it is 2010, pp.784-792. [7] J. Yang, et al. "Image super-resolution via sparse representation." IEEE understandable that artifacts and distortions shown in GAN SR transactions on image processing, 2010, pp.2861-2873. predictions can be greatly reduced through CNN-based [8] R. Zeyde, et al. "On single image scale-up using sparse-representations." ensemble learning. As discussed already, ensemble learning Proceedings of the international conference on curves and surfaces, 2010, enlarges the amount of data with diverse features and increase pp.711-730. [9] R. Timofte, et al. "A+: Adjusted anchored neighborhood regression for fast robustness accordingly. On the other hand, because most super-resolution." Proceedings of the Asian conference on computer vision, artifacts and distortions in each GAN SR prediction were 2014, pp.111-126. derived from processed images that were obtained using [10] C. Dong, et al. "Learning a deep convolutional network for image super- different approaches, these artifacts and distortions were resolution." Proceedings of the European conference on computer vision, 2014, pp.184-199. unlikely be the same or very coherent. Through ensemble [11] C. Dong, et al. "Image super-resolution using deep convolutional learning, these artifacts and distortions will be either effectively networks." IEEE transactions on pattern analysis and machine intelligence, canceled out or greatly reduced. This ability of alleviating 2015, pp.295-307. artifacts and distortions is a main advantage of our method [12] C. Dong, et al. "Accelerating the super-resolution convolutional neural network." Proceedings of the European conference on computer vision, compared with other state-of-the-art SR methods. 2016, pp.391-407. Another clear advantage of our proposed method is that our [13] J. Kim, et al. "Accurate image super-resolution using very deep results preserve details well. Assuming all subtle features are in convolutional networks." Proceedings of the IEEE conference on computer HR images, and these features are blurred or weakened in LR vision and pattern recognition, 2016, pp.1646-1654. [14] J. Kim, et al. "Deeply-recursive convolutional network for image super- images. After the processing step, in each processed image, resolution." Proceedings of the IEEE conference on computer vision and there are some features strengthened and other features further pattern recognition, 2016, pp.1637-1645. weakened. For example, in Case 1 in Fig. 3, edges were [15] W. Shi, et al. "Real-time single image and video super-resolution using an strengthened through ZIP, and texture was strengthened by A+. efficient sub-pixel convolutional neural network." Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.1874- Once most features are strengthened by at least one of the processing algorithms, it is in principle possible to collect a [16] W. Lai, et al. "Fast and accurate image super-resolution with deep relatively complete feature set and recover all details through laplacian pyramid networks." IEEE transactions on pattern analysis and ensemble learning. As a result, the final SR images after machine intelligence, 2018. [17] C. Ledig, et al. "Photo-realistic single image super-resolution using a ensemble learning are with clear details, outperforming generative adversarial network." Proceedings of the IEEE conference on individual model-based methods. computer vision and pattern recognition, 2017, pp.4681-4690. Compared to other model-based methods, our method is [18] X. Wang, et al. "ESRGAN: Enhanced super-resolution generative featured by ensemble learning that combines multiple neural adversarial networks." Proceedings of the European conference on computer vision, 2018. network models together, but it demands a higher [19] Y. Yuan, et al. "Unsupervised image super-resolution using cycle-in-cycle computational cost. generative adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.701-710. V. CONCLUSION [20] C. You, et al. "CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE)." arXiv preprint We have proposed an ensemble learning based SR method to 2018, arXiv:1808.04256. improve the image quality of MR images. We have first used 5 [21] B. Lim, et al. "Enhanced deep residual networks for single image super- resolution." Proceedings of the IEEE conference on computer vision and existing SR algorithms to enlarge LR images and obtained PLR pattern recognition, 2017, pp.136-144. images with complementary priors. Then, a GAN framework [22] T. Tong, et al. "Image super-resolution using dense skip connections." has been used to produce SR prediction results based on Proceedings of the IEEE international conference on computer vision, different PLR images. Next, we can obtain superior SR images 2017, pp.4799-4807. [23] Y. Zhang, et al. "Residual dense network for image super-resolution." through a CNN-based ensemble learning. The SR images Proceedings of the IEEE conference on computer vision and pattern obtained after ensemble learning is better than any individual recognition, 2018, pp.2472-2481. PLR images or GAN SR prediction results. Compared with [24] Y. Zhang, et al. "Image super-resolution using very deep residual channel other state-of-the-art methods, our method enjoys advantages attention networks." Proceedings of the European conference on computer vision, 2018, pp.286-301. of minimal artifacts, little distortions, insignificant noise, and [25] W. Han, et al. "Image super-resolution via dual-state recurrent networks." rich details. Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.1654-1663. REFERENCES [26] M. S. Sajjadi, et al. "Frame-recurrent video super-resolution." Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, [1] E. Plenge, et al. "Super‐resolution methods in MRI: can they improve the pp.6626-6634. trade‐off between resolution, signal‐to‐noise ratio, and acquisition time?" [27] Y. Huang, et al. "Bidirectional recurrent convolutional networks for multi- Magnetic resonance in medicine, 2012, pp.1983-1993. frame super-resolution." Advances in neural information processing [2] E. V. Reeth, et al. "Super‐resolution in magnetic resonance imaging: a systems, 2015, pp.235-243. review." Concepts in magnetic resonance part a, 2012, pp.306-325. [28] A. S. Chaudhari, et al. "Super‐resolution musculoskeletal MRI using deep [3] S. C. Park, et al. "Super-resolution image reconstruction: a technical learning." Magnetic resonance in medicine, 2018, pp.2139-2154. overview." IEEE signal processing magazine, 2003, pp.21-36. [29] K. Zeng, et al. "Simultaneous single-and multi-contrast super-resolution [4] X. Li, et al. "New edge-directed interpolation." IEEE transactions on image for brain MRI images based on a convolutional neural network." processing, 2001, pp.1521-1527. Computers in biology and medicine, 2018, pp.133-141. [30] Y. Chen, et al. "Efficient and accurate MRI super-resolution using a generative adversarial network and 3D multi-Level densely connected > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 10 network." Proceedings of the international conference on medical image over 100 papers to theory, algorithms, artifact reduction and computing and computer-assisted intervention, 2018, pp.91-99. biomedical applications in this important area of CT research. [31] Y. Chen, et al. "Brain MRI super resolution using 3D deep densely Currently, there are 100+ million medical CT scans yearly with connected neural networks." Proceedings of the 2018 IEEE 15th international symposium on biomedical imaging, 2018, pp.739-742. a majority in the spiral/helical cone-beam/multi-slice mode. Dr. [32] C. Liu, et al. "Fusing multi-scale information in convolution network for Wang’s group developed interior tomography theory and MR image super-resolution reconstruction." Biomedical engineering algorithms to solve the long-standing “interior problem” for online, 2018, pp.114. high-fidelity local reconstruction, enabling omni-tomography [33] C. Zhang, et al. "Ensemble machine learning: methods and applications." Springer Science & Business Media, 2012. (“all-in-one”) with CT-MRI as an example. He initiated the area [34] T. G. Dietterich. "Ensemble methods in machine learning." International of bioluminescence tomography with a significant impact on workshop on multiple classifier systems, 2000, pp.1-15. biophotonics. He has published 460+ journal publications, [35] L. Wang, et al. "Ensemble based deep networks for image super- covering imaging-related diverse topics at depth and receiving resolution." Pattern recognition, 2017, pp.191-198. [36] R. Liao, et al. "Video super-resolution via deep draft-ensemble learning." a high number of citations and academic awards. His results Proceedings of the IEEE international conference on computer vision, were featured in Nature, Science, PNAS, and various news 2015, pp.531-539. media. In 2016, he published the first perspective on deep- [37] M. A. Bernstein, et al. "Effect of windowing and zero‐filled reconstruction learning-based tomographic imaging as the new direction of of MRI data on spatial resolution and acquisition strategy." Journal of magnetic resonance imaging: an official journal of the international machine learning and the Nature Machine Intelligence paper on society for magnetic resonance in medicine, 2001, pp.270-280. the superiority of deep learning over iterative reconstruction. [38] I. Gulrajani, et al. "Improved training of Wasserstein GANs." Advances in His team has been continuously well-funded by federal neural information processing systems, 2017, pp.5767-5777. agencies and major imaging companies, actively translating [39] H. Shan, et al. "3-D convolutional encoder-decoder network for low-dose CT via transfer learning from a 2-D trained network." IEEE transactions deep learning techniques into imaging products. His research on medical imaging, 2018, pp.1522-1534. interests include x-ray CT, MRI, optical tomography, [40] Q. Yang, et al. "Low-dose CT image denoising using a generative multimodality fusion, and machine learning. He is the Lead adversarial network with Wasserstein distance and perceptual loss." IEEE Guest Editor for five IEEE Transactions on Medical Imaging transactions on medical imaging, 2018, pp.1348-1357. [41] J. Zbontar, et al. "fastMRI: An open dataset and benchmarks for Special Issues, Founding Editor-in-Chief of International accelerated MRI." arXiv preprint, 2018, arXiv:1811.08839. Journal of Biomedical Imaging, Board Member of IEEE [42] I. Goodfellow, et al. "Generative adversarial nets." Advances in neural Access, and Associate Editor of IEEE Trans. Medical Imaging information processing systems, 2014, pp.2672-2680. (TMI) (recognized as “Outstanding Associate Editor” by TMI), [43] H. Gudbjartsson, et al. "The Rician distribution of noisy MRI data." Magnetic resonance in medicine, 1995, pp.910-914. Medical Physics, and Machine Learning Science and [44] M. Lin, et al. "Network in network." arXiv preprint, 2013, Technology. He is Fellow of IEEE, SPIE, OSA, AIMBE, arXiv:1312.4400. AAPM, and AAAS. Qing Lyu received his Bachelor of Science degree and Master of Science degree in the department of biomedical engineering in Shanghai Jiao Tong University in 2013 and 2016 respectively. In 2017, he joined Dr. Ge Wang’s lab. His main research interests include the MRI postprocessing and signal optimization. Hongming Shan received the B.S. degree from Shandong University of Technology, China, in 2011, and the Ph.D. degree from Fudan University, China, in 2017. He is currently a Postdoctoral Scholar with the Rensselaer Polytechnic Institute, Troy, New York, USA. His research interests include machine/deep learning, computer vision, dimension reduction, and biomedical imaging. Ge Wang is the Clark & Crossan Endowed Chair Professor and Director of the Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY, USA. He published the first spiral/helical cone- beam/multi-slice CT algorithm in 1991 and has since then systematically contributed

Journal

Electrical Engineering and Systems Science – arXiv (Cornell University)

Published: Jul 6, 2019

There are no references for this article.