Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Automated diagnosis of COVID-19 with limited posteroanterior chest X-ray images using fine-tuned deep neural networks

Automated diagnosis of COVID-19 with limited posteroanterior chest X-ray images using fine-tuned... The novel coronavirus 2019 (COVID-19) is a respiratory syndrome that resembles pneumonia. The current diagnostic procedure of COVID-19 follows reverse-transcriptase polymerase chain reaction (RT-PCR) based approach which however is less sensitive to identify the virus at the initial stage. Hence, a more robust and alternate diagnosis technique is desirable. Recently, with the release of publicly available datasets of corona positive patients comprising of computed tomography (CT) and chest X-ray (CXR) imaging; scientists, researchers and healthcare experts are contributing for faster and automated diagnosis of COVID-19 by identifying pulmonary infections using deep learning approaches to achieve better cure and treatment. These datasets have limited samples concerned with the positive COVID-19 cases, which raise the challenge for unbiased learning. Following from this context, this article presents the random oversampling and weighted class loss function approach for unbiased fine-tuned learning (transfer learning) in various state-of-the-art deep learning approaches such as baseline ResNet, Inception-v3, Inception ResNet-v2, DenseNet169, and NASNetLarge to perform binary classification (as normal and COVID-19 cases) and also multi-class classification (as COVID-19, pneumonia, and normal case) of posteroanterior CXR images. Accuracy, precision, recall, loss, and area under the curve (AUC) are utilized to evaluate the performance of the models. Considering the experimental results, the performance of each model is scenario dependent; however, NASNetLarge displayed better scores in contrast to other architectures, which is further compared with other recently proposed approaches. This article also added the visual explanation to illustrate the basis of model classification and perception of COVID-19 in CXR images. Keywords COVID-19 · Classification · Deep learning · Transfer learning · Pneumonia · Chest X-ray (CXR) · Imbalanced learning 1 Introduction syndrome (MERS). The novel coronavirus disease 2019 (COVID-19), began as an outbreak from epicentre Wuhan, Coronaviruses are a large family of viruses that can cause People’s Republic of China in late December 2019, and severe illness to the human being. The first known severe till April 15, 2020, it caused 1,996,681 infections and epidemic is severe acute respiratory syndrome (SARS) 127,590 deaths worldwide [58]. The coronavirus (COVID- occurred in 2003, whereas the second outbreak began in 19) outbreak was declared a public health emergency of 2012 in Saudi Arabia with the middle east respiratory international concern by WHO on January 30, 2020 [59]. On March 11, as the number of COVID-19 cases has increased thirteen times apart from China with more than This article belongs to the Topical Collection: Artificial Intelli- 118,000 cases in 114 countries and over 4,000 deaths, WHO gence Applications for COVID-19, Detection, Control, Prediction, and Diagnosis declared this a pandemic [38]. Globally, many researchers of medicine, clinical and artificial intelligence areas are Narinder Singh Punn trying hard to mobilize preventive action plans for COVID- [email protected] 19 with identified research priorities. Since this disease is Sonali Agarwal highly contagious, the most desirable preventive measure [email protected] is to identify the infected people to control the spread. Unfortunately, there is no well-known treatment available IIIT Allahabad, Prayagraj 211015, India to cure COVID-19, therefore the identified infected person 2690 N. S. Punn and S. Agarwal must be kept in isolation to break the transmission chain several deep-learning based COVID-19 detection tech- as this patient may become the source of community niques have been proposed [20, 55, 56]. Linda et al. [55] transfer. Till now, the testing kit is the only available option introduced a deep CNN, named COVID-Net for the detec- for diagnosis of COVID-19. Unavailability of testing kits tion of COVID-19 cases from the chest X-ray images. due to excessive demand all over the world is a severe Shuaietal. [56] achieved accuracy, specificity and sensi- problem in the mission against this pandemic. Though tivity of 89.5%, 88% and 87% respectively for COVID-19 several healthcare organizations are claiming for successful identification using CT images. development of testing kits, there is a huge gap in demand There are many datasets available about chest X-rays and supply. The healthcare agencies have accelerated the for the detection of pneumonia [8, 18, 32, 50]; but in rate of development of low-cost testing kits, but the inability present research work, COVID-19 X-ray chest images, [8] to diagnose at early-stage and due to exponential growth of and Radiological Society of North America (RSNA) COVID-19 cases, medical professionals are bound to rely images [50] are utilized to generate all possible samples on other diagnostic measures. of chest infection and also to make the study comparable Clinical studies have shown that most COVID-19 with other research works. The number of COVID-19 patients suffer from lung infection [13]. Although chest infected samples present in this dataset is very limited that CT is a more effective imaging technique for lung-related may lead to biased outcome, hence the objective of this disease diagnosis; CXR is preferred because it is widely research is to maximize the learning ability in presence of available, faster and cheaper than CT. Since COVID-19 a small set of positive class samples. For early diagnosis of infection attacks the epithelial cells that line our respiratory COVID-19, this article presents the effectiveness of random tract, hence X-rays images can be used to analyse oversampling and weighted class loss function approaches the lungs to diagnose pneumonia, lung inflammation, for unbiased fine-tuned learning (transfer learning) in abscesses, and/or enlarged lymph nodes [2]. Due to its easy various state-of-the-art deep learning techniques. transmission, developing techniques to accurately and easily Rest of the manuscript is organized as follows: recent identify the presence of COVID-19 and distinguish it from research articles are discussed in Section 2, and Section 3 other forms of flu and pneumonia is crucial. briefs the dataset. Section 4 contains the proposed method- Biomedical image analysis (segmentation and classifica- ology followed by the evaluation metrics in Section 5. tion) is an admired area of research to make the healthcare Results are discussed in Section 6 whereas the last section system more promising [11]. In this area, advancement contains concluding remarks. in computing infrastructure makes it possible to deploy the deep learning techniques for complex medical image analysis tasks. Recent works have shown that the chest 2 Related work X-rays of patients suffering from COVID-19 depicts cer- tain abnormalities in the radiography [39]. For medical Due to the ample availability of X-ray machines, disease image analysis; deep learning techniques, specifically, con- diagnosis using CXR images are widely used by healthcare volutional neural networks (CNN) are very effective and experts. In case of any suspect of COVID-19; instead of efficient in feature extraction and learning, hence becom- using test kits, an alternate way to detect pneumonia from ing the most popular choice among researchers [25]. CNNs the CXR images is required, so that further investigation have been successfully deployed in the analysis of video can be narrowed down for COVID-19 identification. Many endoscopy [12] and CT images, and also used for the diag- studies have been performed on similar ground with nosis of pediatric pneumonia via chest X-ray images [6, several CXR datasets for diagnosis of pneumonia and other 22]. Chouhan et al. [7] proposed a transfer learning based complications [8, 18, 32, 50]. These studies also advocate deep network approach pre-trained on ImageNet [9]for the need of an automated system for quick diagnosis, pneumonia detection. Wang et al. [57] proposed a cus- because the manual methods of X-ray analysis are time tomized VGG16 model for lung regions identification to consuming and unable to serve the purpose due to limited classify different types of pneumonia. Later, Ronneburger availability of X-ray machine operators or radiologists. et al. [42] demonstrated the effectiveness of image aug- Amid the COVID-19 outbreak, many companies at the mentation with CNN in the presence of a small set of global level around the world embraced a flurry of Artificial images. In the area of biomedical image classification, Intelligence (AI) based solutions to detect COVID-19 on Rajpurkar et al. [40] proposed a dense CNN with 121-layers chest X-ray scans. It is evident that deep learning tools to detect several pathologies including pneumonia using are effectively used to screen mild cases, triage new chest X-rays. Lakhani et al. [26] obtained an area under the infections, and monitor disease advancements. This way of curve (AUC) of 0.95 in pneumonia detection using AlexNet diagnosis can reduce the growing burden on radiologists, and GoogLeNet along with image augmentation. Recently, and also supplant standard nucleic acid tests as the primary Automated diagnosis of COVID-19... 2691 diagnostic tool for coronavirus infection. It is also reported is used to measure accuracy, false positive rate, F1 score, that a swab test needs isolation for testing procedure, Matthew’s correlation coefficient (MCC) and kappa. It whereas chest X-ray based detection can be easily is found that ResNet50 in combination with SVM is manageable. Kermany et al. [22] proposed CXR image- statistically superior when compared to other models. Later, based deep learning model to detect pneumonia and classify Bukhari et al. [4] also used ResNet-50 CNN architectures other diseases using different medical datasets with testing on 278 CXR images, partitioned under 3 groups as accuracy of 92.80%. In another similar research, Stephen normal, pneumonia and COVID-19. This approach gave et al. [51] illustrated an efficient deep learning approach promising results and indicated substantial differentiation for pneumonia classification, by using four convolutional of pulmonary changes caused by COVID-19 from the other layers and two dense layers in addition to classical image types of pneumonia. augmentation and achieved 93.73% testing accuracy. Later, Recently, in another research work, an improved ResNet- Saraiva et al. [44] experimented convolutional neural 50 CNN architecture named COVIDResNet has been pro- networks to classify images of childhood pneumonia by posed [10], where conventional ResNet-50 model is applied using a deep learning model with seven convolutional layers with different training techniques including progressive along with three dense layers while achieving 95.30% resizing, cyclical learning rate finding, and discriminative testing accuracy. Liang and Zheng [27] demonstrated a learning rates to gain fast and accurate training. The exper- transfer learning method with a deep residual network for iment is performed through progressively re-sizing of input pediatric pneumonia diagnosis with 49 convolutional layers images to 128 × 128 × 3, 224 × 224 × 3 and 229 × 229 × 3 and two dense layers and achieved 96.70% testing accuracy. pixels, and automatic learning rate selection for fine-tuning In similar research, Wu et al. [60] focused on convolutional the network at each stage. This work claimed to be com- deep neural learning networks and random forest to propose putationally efficient and highly accurate for multi-class a pneumonia prediction using CXR images and achieved classification. 97% testing accuracy. A new deep anomaly detection model is developed by Afterwards, Narin et al. [31] proposed a deep convo- Zhang et. al. [62] for fast and more reliable screening. lutional neural network based automatic prediction model To evaluate the model performance, CXR image data of of COVID-19 with the help of pre-trained transfer models COVID-19 cases and other pneumonia has been collected using CXR images. In this research, authors used ResNet50, from two different sources. To eliminate the data imbalance InceptionV3 and Inception-ResNetV2 pre-trained models problem in the collected samples, authors proposed a to obtain a higher prediction accuracy for a subset of CXR based COVID-19 screening model through anomaly X-ray dataset. Apostolopoulos et. al. [3] in their study, detection task [35]. utilised state-of-the-art convolutional neural network archi- Following this context, this article proposes to contribute tectures for classifying the CXR images. Transfer Learning for early diagnosis of COVID-19 using the state-of-the-art was adopted to handle various abnormalities present in deep learning architectures, assisted with transfer learning the dataset. Two datasets from different repositories have and class imbalance learning approaches. been used to study images of three classes: COVID-19, bacterial/viral pneumonia and normal condition. The arti- cle establishes the suitability of the deep learning model 3 Dataset description with the help of accuracy, sensitivity, and specificity parameters. In this research three datasets are utilized for experiments: In another research, generative adversarial networks COVID-19 image [8], Radiological Society of North (GAN) are used by Khalifa et al. [23] to detect pneumonia America (RSNA) [50] and U.S. national library of from CXR images. The authors addressed the overfitting medicine (USNLM) collected Montgomery country - problem and claimed its robustness by generating more NLM(MC) [19]. COVID-19 image dataset is a public images through GAN. The dataset containing 5863 CXR database of pneumonia cases with CXR images related to images of two categories: normal and pneumonia, has been COVID-19, MERS, SARS, and ARDS collected by Cohen used with typical deep learning models such as AlexNet, et al. [8] from multiple resources available at public domains GoogLeNet, Squeeznet and Resnet18 to detect pneumonia. without infringing patient’s confidentiality (Fig. 1b). It is This research highlights that the Resnet18 outperformed claimed that this dataset can help to identify characteristics among other deep transfer models in combination with of COVID-19 in contrast to other types of pneumonia; GAN. Further, Sethy et al. [46] proposed a deep learning therefore it can play a major role in predicting survival rate. based model to identify coronavirus infections using CXR The dataset includes the statistics up to March 25, 2020 images. Deep features from CXR images have been consisting of 5 types of pneumonia such as SARSr-CoV-2 extracted and support vect or machine (SVM) classifier or COVID-19, SARSr-CoV-1 or SARS, Streptococcus spp., 2692 N. S. Punn and S. Agarwal Fig. 1 Sample chest radiographs Pneumocystis spp. and ARDS with following attributes: 4 Proposed contribution patient ID, offset, sex, age, finding, survival, view, modality, date, location, filename, doi, url, license, clinical notes, and The era of artificial intelligence has brought significant other notes. improvements in the living society [30]. The recent Another dataset utilized in this study is published under advancements in deep learning have extended its domain RSNA pneumonia detection challenge is a subset of 30,000 in various applications such as healthcare, pixel restoration, examinations taken from the NIH CXR14 dataset [50]. visual recognition, signal processing, and a lot more [28]. Out of 30,000 selected images, 15,000 examinations had In healthcare domain, the deep learning based image positive cases of pneumonia and from the remaining 15000 processing approaches for classification and segmentation cases, 7500 cases had no findings and other 7500 cases are applied for faster, efficient, and early diagnosis of the had symptoms other than pneumonia. All these images are deadly diseases e.g. breast cancer, brain tumor, etc. by using annotated by a group of experts including radiologists in two different imaging modalities such as X-ray, CT, MRI, [47] stages. A sample image is shown in Fig. 1c. This dataset and fused modalities [37] along with its future possibilities. has been published in two stages. In Stage one, 25,684 The success of these approaches is dependent on the large training images were considered to test 1,000 images. amount of data availability, which however is not in the case Later in stage two 1000 testing samples were added to the of automated COVID-19 detection. training set to form the dataset of 26,684 training images The main contribution of the work is divided into and a new set of 3,000 radiographs were introduced for the components as shown in Fig. 2. It has two concrete the test. For robust testing and comprehensive coverage components: data preprocessing and classification. COVID- of the comparative analysis, NLM(MC) [19] dataset is 19 image, RSNA and NLM(MC) datasets are used to also utilized that consists of 138 chest posterior-anterior x- generate the final working set. The newly generated rays samples of tuberculosis and normal cases. A sample dataset contains CXR images of the following classes: image is represented in Fig. 1d. Table 1 presents the class coronavirus caused diseases, pneumonia, other diseases summary details of the fused dataset resulting from the and normal cases. Further, binary classification (COVID- above discussed datasets which is utilized for training, 19 vs others) and multi-class classification (COVID- testing and validation of the proposed approach. The fused 19, other types of pneumonia, tuberculosis and normal) dataset is composed of 1214 posteroanterior chest x-ray are achieved using random oversampling and weighted samples with classes labeled as COVID-19 (108), other class loss function approaches for unbiased fine-tuned pneumonia (515), tuberculosis (58) and normal (533). The learning (transfer learning) in various state-of-the-art deep generated fused dataset is publicly available [36]. learning approaches such as baseline ResNet, Inception-v3, Inception ResNet-v2, DenseNet169, and NASNetLarge [17, 52, 54, 63]. The trained models are utilized for identification and classification of COVID-19 in novel samples. Later, Table 1 Fused dataset disease class summary details visualization techniques are utilized to understand and Dataset Findings PCXR images Total samples elaborate the basis of the classification results. COVID-19 COVID-19 108 153 4.1 Imbalanced learning approach Other pneumonia 45 RSNA Normal 453 923 Class balancing techniques are necessary when minority Other pneumonia 470 classes are more important. The dataset used in this research NLM(MC) Tuberculosis 58 138 is highly imbalanced which may lead to biased learning of Normal 80 the model. Number of coronavirus infected CXR images are Automated diagnosis of COVID-19... 2693 Fig. 2 Schematic representation of the proposed components for COVID-19 identification very less compared to other classes, hence class balancing statistical operations. In this work, the samples of CXR image of COVID-19 positive cases are less as compared techniques must be insured to smoothen the learning to other classes, therefore, these minority class images are process. This section discusses two approaches to handle randomly oversampled by means of rotation, scaling, and the class imbalance problem: weight class approach and displacement with the objective to achieve equal distribution random oversampling [21]. of classes and accommodate unbiased learning among the deep learning models. 4.1.1 Weighted class approach 4.2 Classification Strategy In this approach, the intention is to balance the data by altering the weights that each training sample class carries Based on the type of data samples availability of CXR when computing the loss. Normally, each class carries images the COVID-19 classification is divided into two equal weights, but sometimes certain classes with minority following schemes: samples are required to hold more weights if they are more important because training examples within that class – Binary Classification - In this classification scheme, the should have a significant effect on the loss function. In the coronavirus positive samples labelled as “1” (COVID- used dataset the coronavirus infected image class samples 19) are identified against the rest of the samples labelled must be given more weights as they are more significant. In as “0” (non COVID-19 case) which involves other cases this article, the weights for each class is generated based on e.g. chlamydophila, SARS, streptococcus, tuberculosis, the Eq. (1). etc., along with the normal cases. – Multi-class Classification—In this classification scheme, the aim is to distinguish and identify the c=0 w(c) = C . (1) COVID-19 samples from the other pneumonia cases N .n along with the presence of tuberculosis and normal case where C is the class constant for a class c, N is the number c findings. The multi-class classification is performed of classes, and n is the number of samples in a class c. The with three and four classes. The three classes are pro- computed class weights are later fused with the objective vided with labels as “0” being a normal case, “1” being function (loss function) of the deep learning model in order a COVID-19 case, and “2” being other pneumonia and to heavily penalize the false predictions concerned with the tuberculosis cases, whereas four classes are labeled minority samples, which in this case is coronavirus. as “0” being a normal case, “1” being a COVID-19 case, and “2” being other pneumonia case and “3” as 4.1.2 Random oversampling approach tuberculosis case. In both the classification strategies, the deep learning In this approach, the objective is to increase the number models are trained with the above discussed imbalanced of minority samples by utilizing the existing samples learning approaches using the weighted categorical cross belonging to the minority class. The minority samples entropy (WCE) loss function as given by Eq. (2)and are increased until the samples associated with every Eq. 3 [24]: class become equal. Hence the procedure follows by identifying the difference between the number of samples in majority and minority class. To fill this void of difference, f(s) = (2) the samples are generated from the randomly selected sample belonging to the minority class by applying certain C 2694 N. S. Punn and S. Agarwal where i(x,y) is an input image, max and min are max th th WCE =− w(i).t .log(f (s) ) (3) i i and min thresholds to design the mask. Despite filtering the unwanted information, there is still the possibility of In categorical cross entropy, the distribution of the uncertainty at the deep pixel level representation [15]. The predictions (the activations in the output layer, one for each denoising or removal of such uncertainty is carried through class) is compared with the true distribution only, to ensure the adaptive total variation method [53] while preserving the the clear representation of the true class as one-hot encoded original distribution of pixel values. vector; here, closer the model’s outputs are to that vector, Let for a given grayscale image f, on a bounded set  over 2 2 the lower the loss. R ,where  ⊂ R , denoising image u that closely matches to observed image x = (x ,x )  -pixels, givenas 1 2 4.3 Data Preprocessing u = arg min (u − f .ln u)dx + (ω(x)| u|dx) In this article, due to the limited samples of posteroanterior chest X-ray images concerned with positive COVID- (5) 19 [8] cases, the data samples are mixed with the other randomly selected CXR images selected from other where ω(x) = , G - the Gaussian kernel 1+k mod G ∗u datasets-, RSNA [50] and NLM(MC) [19]. The RSNA for smoothing with σ variance, k > 0 is contrast parameter and NLM(MC) datasets consists of posteroanterior CXR and * is convolution operator. images covering sample cases labelled as pneumonia Figure 3 illustrates the data preprocessing stages by and tuberculosis respectively along with normal samples. considering an instance of COVID-19 case consisting of Table 2 describes the distribution of training, testing, and textual and symbolic artifacts from the generated dataset. validation sets using the fused dataset for binary and multi- The resulting distributed pixels histograms at each stage class classification along with different class imbalance of preprocessing shown in Fig. 3, illustrates that the strategies i.e. class weighted loss function that penalizes preprocessing approach tends to preserve the original nature the model for any false negative prediction and random of distribution of the pixels while removing the irregular oversampling [61] of minority classes which in this case is intensities. The preprocessed images are then divided COVID-19. into training, testing, and validation set for training and The CXR images in the aggregated dataset also consists evaluation of the state-of-the-art deep learning classification of unwanted artifacts such as bright texts, symbols, varying models. resolutions and pixel level noise, which necessitates its preprocessing. In order to suppress the highlighted textual 4.4 Deep learning models and symbolic noise, the images are inpainted with the image mask generated using binary thresholding [34]asgiven by This section incorporates the state-of-the-art deep learning Eq. (4), followed by resizing the images to a fixed size models utilized in the present research work as shown resolution of 331 × 331 × 3. in Table 3 along with their respective contribution, max th, i(x, y) ≥ min th. parameters, and performance on the standard benchmark M(x, y) = (4) 0, otherwise. datasets. The inception deep convolutional architectures Table 2 Posteroanterior CXR images distribution into training, validation, and test sets from the fused datasets for different problem definitions Findings Class imbalanced learning strategy Classification with class weighted loss function Classification with random oversampling Binary Multi-class Multi-class Binary Multi-class Multi-class (2 classes) (3 classes) (4 classes) (2 classes) (3 classes) (4 classes) Tr Val Tst Tr Val Tst Tr Val Tst Tr Val Tst Tr Val Tst Tr Val Tst Normal 906 90 110 437 44 52 437 44 52 960 90 110 469 44 52 437 44 52 Tuberculosis 469 46 58 47 5 6 469 46 58 437 5 6 Other pneumonia 422 41 52 437 41 52 COVID-19 88 9 11 88 9 11 88 9 11 960 9 11 469 9 11 437 9 11 Total 994 99 121 994 99 121 994 99 121 1920 99 121 1407 99 121 1748 99 121 Automated diagnosis of COVID-19... 2695 Fig. 3 Data preprocessing stages of raw posteroanterior CXR image proposed by GoogLeNet are considered as the state-of- Later, Inception-ResNet-v2 was proposed by Szegedy et al. the-art deep learning architectures for image analysis and [52]. This hybrid model is a combination of residual object identification with the basic model as inception- connections and a recent version of Inception architecture. v1 [5]. Later, this base model was refined by introducing It is intended to train very deep convolutional models by the batch normalization and established as the inception- the additive merging of signals, both for image recognition v2 [54]. In further iterations, additional factorisation was and object detection. This network is more robust and learns introduced and released as the inception-v3. It is one of the rich feature representations. Afterwards, DenseNet was pre-trained models to perform two types of specific tasks: proposed by Huang et al. [17]. It works on the concept of dimensionality reduction using CNN and classification reuse, in which each layer receives inputs from all previous using fully-connected and softmax layers. Since it is layers and yields a condensed model to pass its own feature- originally trained on over a million images consisting of maps to all subsequent layers. This makes the network 1,000 classes of ImageNet, its head layers can be retrained thinner and compact with the fewer number of channels, for the generated dataset using historical knowledge to while improving variation in the input of subsequent layers, reduce the extensive training and computational power. and becomes easy to train and highly parameter efficient. Table 3 Recent deep learning Architecture Year Contribution Param. Dataset E. rate architectures that are reported with top 5 error rate per year Inception-v3 2015 Avoided the bottleneck 23.6M ILSVRC 5.6 [54] representations, Dimension reduction promotes faster learning. Inception 2016 Focused on residual 56M ILSVRC 4.9 ResNet-v2 connection rather [52] than filter connection approach. DenseNet169 2017 Dense connection blocks 17.8M CIFAR-10 5.19 [17] for activation flow across CIFAR-100 19.64 the layers. CIFAR-10+ 3.46 CIFAR-100+ 17.18 NASNetLarge 2018 Search and transfer the 22.6M CIFAR-10 2.4 [63] architecture block from small dataset to large dataset, Scheduled drop path regularization. 2696 N. S. Punn and S. Agarwal Google Brain research team proposed NASNet model based its trainable parameters along with the addition of fully on reinforcement learning search methods [63]. It creates connected layers with 128 neurons and 2 or 3 neurons search space by factoring the network into cells and further in the output layer depending on binary or multi-class dividing it into a number of blocks. Each block is supported classification, accompanied with the softmax activation by a set of popular operations in CNN models with various function. kernel size e.g: convolutions, max pooling, average pooling, dilated convolution, depth-wise separable convolutions etc. 5 Evaluation metrics 4.5 Fine tuning The deep learning models are trained using the train- Training deep learning models require (Inception-v3, ing and validation set under consideration of each pos- InceptionResNet-v2, etc.) exhaustive amount of resources sible combination of the discussed approaches of class and time. While these networks, attain relatively excellent imbalance learning and classification strategy, thereby mak- performance on ImageNet [9], training them on a CPU is ing four possible scenarios for training a model. Later, an exercise in futility. These CNNs are often trained for the trained models are evaluated on the test set using a couple of weeks or more using arrays of GPUs to get the standard benchmark performance metrics such as good results on the complex and complicated datasets. In accuracy, precision, recall (selectivity), area under curve most deep CNNs the first few convolution layers learn (AUC), specificity and F1 score (dice coefficient) as shown low-level features (edges, curves, blobs) and with progress, in Fig. 5. through the network, it learns more mid/high-level features or patterns associated with the on-going task. In fine tuning, the aim is to keep or freeze these trained low-level features, 6 Training and testing and only train the high-level features needed for our new image classification problem. The proposed approach is trained and tested on the above In this article, the trials of training the deep learning discussed fused dataset using mini-batch gradient descent classification models initiates with the baseline residual with learning rate optimizer as adam [43]via N-Vidia network that is composed of five residual blocks [16], Titan GPU. The training, testing, and validation sets are defined by two convolutions whose rectified linear unit highlighted in Table 2. The training phase is assisted (ReLU) activation is concatenated with the input of the first with the batch size of 10, 4-fold cross validation, switch convolution layer, followed by max-pooling and instance normalization [29] to adapt to instance or batch or layer normalization, except the final output layer which uses normalization, and earlystopping technique that aims to stop softmax activation function. Later, transfer learning is the training process if the validation error stops decreasing. utilized on the state-of-the-art architectures discussed in For each epoch the above discussed evaluation metrics are Table 3 to utilize the pre-trained models while fine tuning computed on the training and validation set to analyse the the head layers to serve the purpose of classifying the model’s performance and improve the classification results. COVID-19 samples, as illustrated in Fig. 4. This follows Later, the test set is utilized to evaluate the results of the by enabling the four head layers of the network to adjust proposed approach. Fig. 4 Schematic representation of training framework for deep learning architectures via transfer learning Automated diagnosis of COVID-19... 2697 Fig. 5 Confusion matrix and performance evaluation metrics 7 Results and discussion the training phase for classification of COVID-19 samples, generated using the tensorboard. Since the models are s With extensive trials, it is observed that there is no initialized with the trained weights on the ImageNet and individual model that displayed best performance for all the head layers are fine-tuned for classification, the training of scenarios in terms of accuracy, precision, recall, specificity, the models initiates with descent metrics values. The best AUC, and F1-score. However, the NASNetLarge mostly scores of the models corresponding to each scenario are claimed the best scores, thus making it best fit for the highlighted in Table 4 along with the detailed comparative classification of COVID-19 samples. It is also observed analysis of the results for classifying the COVID-19 samples that the results of binary classification (COVID-19 vs non- in terms of discussed evaluation metrics for different COVID-19) are better than the multi-class classification scenarios CB, CM ,CM RB, RM and RM where C and 3 4 3 4 (COVID-19 vs other classes). With this it is evident that R indicates class imbalanced learning approaches, namely by grouping other diseases together as non-COVID-19 weighted class loss functions and random oversampling samples, models can efficiently learn features and patterns of the minority class, whereas B indicates the binary belonging to the COVID-19. Figure 6 shows the average classification and M (3 classes), M (4 classes) indicate 3 4 performance curves of the NASNetLarge model evaluated multi-class classification schemes. Furthermore, the results and monitored on the test set for each iteration during obtained are compared with the other recently proposed Fig. 6 Monitoring every epoch of training with performance curves for the NASNetLarge model with average a accuracy, b AUC, c specificity, d F1-score, e precision, and f recall. The curves are smoothed using moving exponential average 2698 N. S. Punn and S. Agarwal Table 4 Comparative analysis Model Scenario Accuracy Precision Recall AUC Specifcity F1-score of the classification results of the deep learning models for Baseline ResNet CB 0.84 0.67 0.89 0.90 0.85 0.76 each scenario CM 0.77 0.62 0.89 0.90 0.77 0.73 CM 0.77 0.61 0.83 0.90 0.77 0.70 RB 0.89 0.56 0.56 0.89 0.60 0.56 RM 0.76 0.63 0.42 0.83 0.77 0.50 RM 0.76 0.65 0.43 0.82 0.73 0.52 Inception v2 CB 0.88 0.77 0.79 0.93 0.89 0.78 CM 0.87 0.73 0.88 0.90 0.88 0.80 CM 0.87 0.73 0.85 0.89 0.85 0.79 RB 0.88 0.78 0.70 0.88 0.91 0.74 RM 0.87 0.90 0.81 0.91 0.90 0.85 RM 0.84 0.89 0.81 0.91 0.88 0.85 Inception ResNet v2 CB 0.95 0.86 0.66 0.98 0.87 0.75 CM 0.92 0.85 0.92 0.99 0.89 0.88 CM 0.91 0.83 0.91 0.98 0.89 0.87 RB 0.90 0.85 0.75 0.93 0.88 0.80 RM 0.92 0.97 0.96 0.99 0.94 0.96 RM 0.91 0.95 0.93 0.98 0.93 0.94 DenseNet 169 CB 0.90 0.82 0.78 0.91 0.91 0.80 CM 0.87 0.87 0.91 0.95 0.89 0.89 CM 0.86 0.87 0.89 0.93 0.89 0.88 RB 0.95 0.94 0.96 0.97 0.95 0.95 RM 0.91 0.93 0.95 0.93 0.95 0.94 RM 0.92 0.94 0.95 0.94 0.95 0.94 NASNet Large CB 0.97 0.82 0.91 0.99 0.98 0.86 CM 0.94 0.89 0.91 0.94 0.89 0.90 CM 0.95 0.88 0.89 0.92 0.89 0.88 RB 0.98 0.87 0.90 0.99 0.94 0.98 RM 0.96 0.93 0.91 0.96 0.94 0.92 RM 0.95 0.95 0.90 0.95 0.94 0.92 Bold numbers highlight the best scores of the models corresponding to each scenario approaches to detect COVID-19 positive symptoms via X- proposed approach outperforms other approaches, whereas ray images as shown in Table 5. It is observed that the Apostolopoulos et al. [3] achieved similar results with the Table 5 Comparative analysis of the proposed approach with recently proposed strategies Authors Technique Network Classes Accuracy Precision Recall AUC Specificity Yujin et al. [33] Fine tuning Res- Net18 4 0.88 0.83 0.86 – 0.96 Afshar et al. [1] Fine tuning Capsule 0.95 – 0.90 0.95 Proposed Fine NASNet Large 0.95 0.95 0.90 0.94 0.92 Wang et al. [55] Full training COVID-Net 3 0.92 0.91 0.88 – – Apostolopoulos et al. [3] Fine tuning VGG19 0.87 – 0.92 – 0.98 Proposed Fine tuning NASNet Large 0.96 0.93 0.91 0.96 0.94 Hall et al. [14] Fine tuning Ensemble 2 0.91 – 0.78 0.94 0.93 Apostolopoulos et al. [3] Fine tuning VGG19 0.98 – 0.92 – 0.98 Proposed Fine tuning NASNet Large 0.98 0.87 0.90 0.99 0.98 Bold numbers highlight the results obtained using proposed approach Automated diagnosis of COVID-19... 2699 Fig. 7 LIME explanation of three distinct class samples using NASNetLarge along with the prediction probabilities of sample being normal (N), COVID-19 (C), and other pneumonia (OP) help of VGG19 model [49] but with approximately double There are many visualization techniques for example class trainable parameters (144M) as compared to the proposed activation maps (CAM) [45], saliency maps (SM) [48], local approach (83M). interpretable model-agnostic explanations (LIME) [41], and a lot more. In this article, activation maps and LIME techniques are utilized to present the model perception of 8 Visualization identifying and classifying the COVID-19 samples from CXR images. CAM aims at understanding the feature space Training the model and getting the results is not sufficient of an input image that influences the prediction, whereas unless it is understood that what is triggering the concerned LIME is an innovative explanation technique to represent output. To handle this blackbox, visualization techniques the model prediction with local fidelity, interpretability assist in illustrating the basis of prediction of the model. and model agnostic. For instance, fine-tuned NASNetLarge Fig. 8 CAM of NASNetLarge model for a Normal, b COVID-19, and c Other pneumonia 2700 N. S. Punn and S. Agarwal architecture is considered to generate LIME explanations convolutional neural networks. Phys Eng Sci Med 1:635–640. https://doi.org/10.1007/s13246-020-00865-4 for some samples taken from the test set (see Fig. 7), 4. Bukhari SUK, Bukhari SSK, Syed A, SHAH SSH (2020) The whereas class activation maps tend to present the patterns diagnostic evaluation of convolutional neural network (cnn) for learned by the model for classification of samples as shown the assessment of chest x-ray of patients infected with covid-19. in Fig. 8. Figure 7 presents the LIME technique applied to medRxiv 5. Chen Q, Montesinos P, Sun QS, Heng PA et al (2010) Adaptive four samples belonging to four distinct classes as COVID- total variation denoising based on difference curvature. Image Vis 19, other types of pneumonia, tuberculosis and normal Comput 28(3):298–306 cases. The red and green areas in the LIME generated expla- 6. Choe J, Lee SM, Do KH, Lee G, Lee JG, Lee SM, Seo JB nation correspond to the regions that contributed against the (2019) Deep learning–based image conversion of ct reconstruction kernels improves radiomics reproducibility for pulmonary nodules predicted class and towards the predicted class respectively. or masses. Radiology 292(2):365–373 7. Chouhan V, Singh SK, Khamparia A, Gupta D, Tiwari P, Moreira C, Damase ˇ vicius ˇ R, de Albuquerque VHC (2020) A novel transfer 9 Conclusion learning based approach for pneumonia detection in chest x-ray images. Appl Sci 10(2):559 8. Cohen JP, Morrison P, Dao L (2020) Covid-19 image data This article proposes to leverage the state-of-the-art deep collection. arXiv:2003.11597 learning models to aid in early identification and diagnosis 9. Deng J, Dong W, Socher R, Li L, Kai L, Li F-F (2009) Imagenet: of COVID-19 virus by using the limited posteroanterior a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248– chest X-ray images. Each trained model was evaluated using benchmark performance metrics e.g. accuracy, precision, 10. Farooq M, Hafeez A (2020) Covid-resnet: a deep learn- recall, area under curve, specificity, and F1 score under four ing framework for screening of covid19 from radiographs. different scenarios concerned with imbalanced learning and arXiv:2003.14395 11. Fourcade A, Khonsari R (2019) Deep learning in medical image classification strategy. With extensive trials, it was observed analysis: a third eye for doctors. J Stomatol Oral Maxillofac Surg that models achieve different scores in different scenarios, 120(4):279–288 among which NASNetLarge displayed better performance 12. Gomez ´ P, Semmler M, Schutzenber ¨ ger A, Bohr C, Dollinger ¨ M specially in binary classification of COVID-19 samples. (2019) Low-light image enhancement of high-speed endoscopic videos using a convolutional neural network. Med Biol Eng The visual representation based on local interpretable model Comput 57(7):1451–1463 agnostic explanations is utilized to understand the basis of 13. Guo YR, Cao QD, Hong ZS, Tan YY, Chen SD, Jin HJ, Tan KS, prediction of the model. As an extension to this work more Wang DY, Yan Y (2020) The origin, transmission and clinical deep learning models and preprocessing techniques can be therapies on coronavirus disease 2019 (covid-19) outbreak–an update on the status. Mil Med Res 7(1):1–10 explored to achieve better results. 14. Hall LO, Paul R, Goldgof DB, Goldgof GM (2020) Finding covid- Acknowledgements We are very much obliged to our institute, Indian 19 from chest x-rays using deep learning on a small dataset. Institute of Information Technology Allahabad (IIITA), India and Big arXiv:2004.02060 Data Analytics (BDA) lab for ensuring the needed resources and 15. Hasinoff SW (2014) Photon, poisson noise. Computer Vision: A support. We also would like to extend our thanks to the colleagues Reference Guide, Boston, pp 608–610 for their valuable guidance and suggestions. The manuscript is also 16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for available in the pre-print arXiv with the eprint-2004.11676 under the image recognition. In: Proceedings of the IEEE conference on license Attribution 4.0 International (CC BY 4.0). computer vision and pattern recognition, pp 770–778 17. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of Compliance with Ethical Standards the IEEE conference on computer vision and pattern recognition, pp 4700–4708 Conflict of interest The authors have no conflict of interest to declare. 18. Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, Marklund H, Haghgoo B, Ball R, Shpanskaya K et al (2019) Ethical approval This article does not contain any studies with human Chexpert: a large chest radiograph dataset with uncertainty labels participants or animals performed by any of the authors. and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 590–597 19. Jaeger S, Candemir S, Antani S, Wang ´ YXJ, Lu PX, Thoma G (2014) Two public chest x-ray datasets for computer-aided References screening of pulmonary diseases. Quant Imaging Med Surg 4(6):475 1. Afshar P, Heidarian S, Naderkhani F, Oikonomou A, Plataniotis 20. Joaquin AS (2020) Using deep learning to detect pneumonia KN, Mohammadi A (2020) Covid-caps: a capsule network-based caused by ncov-19 from x-ray images. https://towardsdatascience. framework for identification of covid-19 cases from x-ray images. com/using-deep-learning-to-detect-ncov-19-from-x-ray-images-1 arXiv:2004.02696 a89701d1acd (Online; Accessed 12 April 2020) 2. Allen JN, Davis WB (1994) Eosinophilic lung diseases. Am J 21. Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning Respir Crit Care Med 150(5):1423–1438 with class imbalance. J Big Data 6(1):27 3. Apostolopoulos ID, Mpesiana TA (2020) Covid-19: automatic 22. Kermany DS, Goldbaum M, Cai W, Valentim CC, Liang H, Baxter detection from x-ray images utilizing transfer learning with SL, McKeown A, Yang G, Wu X, Yan F et al (2018) Identifying Automated diagnosis of COVID-19... 2701 medical diagnoses and treatable diseases by image-based deep 42. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional learning. Cell 172(5):1122–1131 networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted 23. Khalifa NEM, Taha MHN, Hassanien AE, Elghamrawy S (2020) intervention. Springer, pp 234–241 Detection of coronavirus (covid-19) associated pneumonia based 43. Ruder S (2016) An overview of gradient descent optimization on generative adversarial networks and a fine-tuned deep transfer algorithms. arXiv:1609.04747 learning model using chest x-ray dataset. arXiv:2004.01184 44. Saraiva A, Ferreira N, Sousa L, Carvalho da Costa N, Sousa 24. Koidl K (2013) Loss functions in classification tasks. School of J, Santos D, Soares S (2019) Classification of images of Computer Science and Statistic Trinity College, Dublin childhood pneumonia using convolutional neural networks. In: 6th 25. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classifi- International conference on bioimaging, pp 112–119 cation with deep convolutional neural networks. In: Advances in 45. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra neural information processing systems, pp 1097–1105 D (2017) Grad-cam: visual explanations from deep networks 26. Lakhani P, Sundaram B (2017) Deep learning at chest radiogra- via gradient-based localization. In: Proceedings of the IEEE phy: automated classification of pulmonary tuberculosis by using international conference on computer vision, pp 618–626 convolutional neural networks. Radiology 284(2):574–582 46. Sethy PK, Behera SK (2020) Detection of coronavirus disease 27. Liang G, Zheng L (2019) A transfer learning method with deep (covid-19) based on deep features. Preprints, 2020030300, 2020 residual network for pediatric pneumonia diagnosis. Computer 47. Shen D, Wu G, Suk HI (2017) Deep learning in medical image Methods and Programs in Biomedicine, pp 104964 analysis. Ann Rev Biomed Eng 19:221–248 28. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A 48. Simonyan K, Vedaldi A, Zisserman A (2013) Deep inside survey of deep neural network architectures and their applications. convolutional networks: visualising image classification models Neurocomputing 234:11–26 and saliency maps. arXiv:1312.6034 29. Luo P, Ren J, Peng Z, Zhang R, Li J (2018) Differ- 49. Simonyan K, Zisserman A (2014) Very deep convolutional entiable learning-to-normalize via switchable normalization. networks for large-scale image recognition. arXiv:1409.1556 arXiv:1806.10779 50. Stein A (2020) Pneumonia dataset annotation methods. rsna pneu- 30. Makridakis S (2017) The forthcoming artificial intelligence (ai) monia detection challenge discussion, 2018. https://www.kaggle. revolution: its impact on society and firms. Futures 90:46–60 com/c/rsna-pneumonia-detection-challenge/discussion/ (Online; 31. Narin A, Kaya C, Pamuk Z (2020) Automatic detection of Accessed 12 April 2020) coronavirus disease (covid-19) using x-ray images and deep 51. Stephen O, Sain M, Maduh UJ, Jeong DU (2019) An efficient convolutional neural networks. arXiv:2003.10849 deep learning approach to pneumonia classification in healthcare. 32. Oakden-Rayner L (2020) Chexnet: an in-depth review. J Healthcare Eng 2019. https://doi.org/10.1155/2019/4180949 https://lukeoakdenrayner.wordpress.com/2018/01/24/ 52. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception- chexnet-an-in-depth-review/ (Online; Accessed 12 April 2020) v4, inception-resnet and the impact of residual connections 33. Oh Y, Park S, Ye JC (2020) Deep learning covid-19 features on on learning. In: Thirty-first AAAI conference on artificial cxr using limited training data sets. IEEE Trans Med Imaging intelligence 39:2688–2700. https://doi.org/10.1109/TMI.2020.2993291 53. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, 34. OpenCV (2020) Image thresholding. https://docs.opencv.org/ Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with master/d7/d4d/tutorial py thresholding.html (Online; Accessed convolutions. In: Proceedings of the IEEE conference on computer 12 April 2020) vision and pattern recognition, pp 1–9 35. Pang G, Shen C, van den Hengel A (2019) Deep anomaly 54. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) detection with deviation networks. In: Proceedings of the 25th Rethinking the inception architecture for computer vision. In: ACM SIGKDD international conference on knowledge discovery Proceedings of the IEEE conference on computer vision and & data mining, pp 353–362 pattern recognition, pp 2818–2826 36. Punn N (2020) Covid-19 posteroanterior chest x-ray fused 55. Wang L, Wong A (2020) Covid-net: a tailored deep convolutional (cpcxr) dataset. https://github.com/nspunn1993/COVID-19-PA- neural network design for detection of covid-19 cases from chest CXR-fused-dataset (Online; Accessed 21 Jul 2020) radiography images. arXiv:2003.09871 37. Punn NS, Agarwal S (2020) Inception u-net architecture for 56. Wang S, Kang B, Ma J, Zeng X, Xiao M, Guo J, Cai M, Yang J, Li semantic segmentation to identify nuclei in microscopy cell Y, Meng X et al (2020) A deep learning algorithm using ct images images. ACM Trans Multimed Comput Commun Appl (TOMM) to screen for corona virus disease (covid-19). MedRxiv 16(1):1–15 57. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers R (2017) 38. Punn NS, Sonbhadra SK, Agarwal S (2020) Covid-19 epidemic Hospital-scale chest x-ray database and benchmarks on weakly- analysis using machine learning and deep learning algorithms supervised classification and localization of common thorax medRxiv diseases. In: IEEE CVPR 39. Rajinikanth V, Dey N, Raj ANJ, Hassanien AE, Santosh K, 58. WHO (2020) Coronavirus disease 2019 (covid-19) situ- Raja N (2020) Harmony-search and otsu based system for ation report–81. https://www.who.int/docs/default-source/ coronavirus disease (covid-19) detection using lung ct scan coronaviruse/situation-reports/. Online; Accessed 12 April 2020 images. arXiv:2004.03431 59. WHO (2020) Who timeline—covid-19. https://www.who.int/ 40. Rajpurkar P, Irvin J, Ball RL, Zhu K, Yang B, Mehta H, Duan news-room/detail/08-04-2020-who-timeline---covid-19 (Online; T, Ding D, Bagul A, Langlotz CP et al (2018) Deep learning Accessed 12 April 2020 for chest radiograph diagnosis: a retrospective comparison of the 60. Wu H, Xie P, Zhang H, Li D, Cheng M (2020) Predict pneumonia chexnext algorithm to practicing radiologists. PLoS Med 15(11): with chest x-ray images based on convolutional deep neural e1002686 learning networks. J Intell Fuzzy Syst (Preprint) 1–15 41. Ribeiro MT, Singh S, Guestrin C (2016) “Why should I 61. Yap BW, Rani KA, Rahman HAA, Fong S, Khairudin Z, Abdullah trust you?” Explaining the predictions of any classifier. In: NN (2014) An application of oversampling, undersampling, Proceedings of the 22nd ACM SIGKDD international conference bagging and boosting in handling imbalanced datasets. In: on knowledge discovery and data mining, pp 1135–1144 2702 N. S. Punn and S. Agarwal Proceedings of the first international conference on advanced data Dr. Sonali Agarwal is and information engineering (DaEng-2013). Springer, pp 13–22 presently working as Asso- 62. Zhang J, Xie Y, Li Y, Shen C, Xia Y (2020) Covid-19 screening on ciate Professor in department chest x-ray images using deep learning based anomaly detection. of information technology arXiv:2003.12338 at IIIT Allahabad, Prayagraj, 63. Zoph B, Vasudevan V, Shlens J, Le QV (2018) Learning U. P., India 211015. She is transferable architectures for scalable image recognition. In: having hands on experience Proceedings of the IEEE conference on computer vision and on stream computing and pattern recognition, pp 8697–8710 complex processing plat- forms such as Apache Spark, Publisher’s note Springer Nature remains neutral with regard to Apache Flink and ESPER. jurisdictional claims in published maps and institutional affiliations. Her research interests are in the areas of Stream Analyt- ics, Big Data, Stream Data Mining, Complex Event Narinder Singh Punn is Processing System, Support presently pursuing his PhD Vector Machines and Software Engineering. from Indian Institute of Information Technology Allahabad, Prayagraj, U. P., India 211015. He is working as a senior research fellow in the “Big Data Analyt- ics” laboratory of IIITA. His research areas are: Machine Learning, Deep Learning and Biomedical Image Analysis. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Intelligence Pubmed Central

Automated diagnosis of COVID-19 with limited posteroanterior chest X-ray images using fine-tuned deep neural networks

Punn, Narinder Singh; Agarwal, Sonali
Applied Intelligence , Volume 51 (5) – Oct 17, 2020
Download PDF
14 pages

Loading next page...
 
/lp/pubmed-central/automated-diagnosis-of-covid-19-with-limited-posteroanterior-chest-x-DrxtWlkdOP

References (116)

Publisher
Pubmed Central
Copyright
© Springer Science+Business Media, LLC, part of Springer Nature 2020
ISSN
0924-669X
eISSN
1573-7497
DOI
10.1007/s10489-020-01900-3
Publisher site
See Article on Publisher Site

Abstract

The novel coronavirus 2019 (COVID-19) is a respiratory syndrome that resembles pneumonia. The current diagnostic procedure of COVID-19 follows reverse-transcriptase polymerase chain reaction (RT-PCR) based approach which however is less sensitive to identify the virus at the initial stage. Hence, a more robust and alternate diagnosis technique is desirable. Recently, with the release of publicly available datasets of corona positive patients comprising of computed tomography (CT) and chest X-ray (CXR) imaging; scientists, researchers and healthcare experts are contributing for faster and automated diagnosis of COVID-19 by identifying pulmonary infections using deep learning approaches to achieve better cure and treatment. These datasets have limited samples concerned with the positive COVID-19 cases, which raise the challenge for unbiased learning. Following from this context, this article presents the random oversampling and weighted class loss function approach for unbiased fine-tuned learning (transfer learning) in various state-of-the-art deep learning approaches such as baseline ResNet, Inception-v3, Inception ResNet-v2, DenseNet169, and NASNetLarge to perform binary classification (as normal and COVID-19 cases) and also multi-class classification (as COVID-19, pneumonia, and normal case) of posteroanterior CXR images. Accuracy, precision, recall, loss, and area under the curve (AUC) are utilized to evaluate the performance of the models. Considering the experimental results, the performance of each model is scenario dependent; however, NASNetLarge displayed better scores in contrast to other architectures, which is further compared with other recently proposed approaches. This article also added the visual explanation to illustrate the basis of model classification and perception of COVID-19 in CXR images. Keywords COVID-19 · Classification · Deep learning · Transfer learning · Pneumonia · Chest X-ray (CXR) · Imbalanced learning 1 Introduction syndrome (MERS). The novel coronavirus disease 2019 (COVID-19), began as an outbreak from epicentre Wuhan, Coronaviruses are a large family of viruses that can cause People’s Republic of China in late December 2019, and severe illness to the human being. The first known severe till April 15, 2020, it caused 1,996,681 infections and epidemic is severe acute respiratory syndrome (SARS) 127,590 deaths worldwide [58]. The coronavirus (COVID- occurred in 2003, whereas the second outbreak began in 19) outbreak was declared a public health emergency of 2012 in Saudi Arabia with the middle east respiratory international concern by WHO on January 30, 2020 [59]. On March 11, as the number of COVID-19 cases has increased thirteen times apart from China with more than This article belongs to the Topical Collection: Artificial Intelli- 118,000 cases in 114 countries and over 4,000 deaths, WHO gence Applications for COVID-19, Detection, Control, Prediction, and Diagnosis declared this a pandemic [38]. Globally, many researchers of medicine, clinical and artificial intelligence areas are Narinder Singh Punn trying hard to mobilize preventive action plans for COVID- [email protected] 19 with identified research priorities. Since this disease is Sonali Agarwal highly contagious, the most desirable preventive measure [email protected] is to identify the infected people to control the spread. Unfortunately, there is no well-known treatment available IIIT Allahabad, Prayagraj 211015, India to cure COVID-19, therefore the identified infected person 2690 N. S. Punn and S. Agarwal must be kept in isolation to break the transmission chain several deep-learning based COVID-19 detection tech- as this patient may become the source of community niques have been proposed [20, 55, 56]. Linda et al. [55] transfer. Till now, the testing kit is the only available option introduced a deep CNN, named COVID-Net for the detec- for diagnosis of COVID-19. Unavailability of testing kits tion of COVID-19 cases from the chest X-ray images. due to excessive demand all over the world is a severe Shuaietal. [56] achieved accuracy, specificity and sensi- problem in the mission against this pandemic. Though tivity of 89.5%, 88% and 87% respectively for COVID-19 several healthcare organizations are claiming for successful identification using CT images. development of testing kits, there is a huge gap in demand There are many datasets available about chest X-rays and supply. The healthcare agencies have accelerated the for the detection of pneumonia [8, 18, 32, 50]; but in rate of development of low-cost testing kits, but the inability present research work, COVID-19 X-ray chest images, [8] to diagnose at early-stage and due to exponential growth of and Radiological Society of North America (RSNA) COVID-19 cases, medical professionals are bound to rely images [50] are utilized to generate all possible samples on other diagnostic measures. of chest infection and also to make the study comparable Clinical studies have shown that most COVID-19 with other research works. The number of COVID-19 patients suffer from lung infection [13]. Although chest infected samples present in this dataset is very limited that CT is a more effective imaging technique for lung-related may lead to biased outcome, hence the objective of this disease diagnosis; CXR is preferred because it is widely research is to maximize the learning ability in presence of available, faster and cheaper than CT. Since COVID-19 a small set of positive class samples. For early diagnosis of infection attacks the epithelial cells that line our respiratory COVID-19, this article presents the effectiveness of random tract, hence X-rays images can be used to analyse oversampling and weighted class loss function approaches the lungs to diagnose pneumonia, lung inflammation, for unbiased fine-tuned learning (transfer learning) in abscesses, and/or enlarged lymph nodes [2]. Due to its easy various state-of-the-art deep learning techniques. transmission, developing techniques to accurately and easily Rest of the manuscript is organized as follows: recent identify the presence of COVID-19 and distinguish it from research articles are discussed in Section 2, and Section 3 other forms of flu and pneumonia is crucial. briefs the dataset. Section 4 contains the proposed method- Biomedical image analysis (segmentation and classifica- ology followed by the evaluation metrics in Section 5. tion) is an admired area of research to make the healthcare Results are discussed in Section 6 whereas the last section system more promising [11]. In this area, advancement contains concluding remarks. in computing infrastructure makes it possible to deploy the deep learning techniques for complex medical image analysis tasks. Recent works have shown that the chest 2 Related work X-rays of patients suffering from COVID-19 depicts cer- tain abnormalities in the radiography [39]. For medical Due to the ample availability of X-ray machines, disease image analysis; deep learning techniques, specifically, con- diagnosis using CXR images are widely used by healthcare volutional neural networks (CNN) are very effective and experts. In case of any suspect of COVID-19; instead of efficient in feature extraction and learning, hence becom- using test kits, an alternate way to detect pneumonia from ing the most popular choice among researchers [25]. CNNs the CXR images is required, so that further investigation have been successfully deployed in the analysis of video can be narrowed down for COVID-19 identification. Many endoscopy [12] and CT images, and also used for the diag- studies have been performed on similar ground with nosis of pediatric pneumonia via chest X-ray images [6, several CXR datasets for diagnosis of pneumonia and other 22]. Chouhan et al. [7] proposed a transfer learning based complications [8, 18, 32, 50]. These studies also advocate deep network approach pre-trained on ImageNet [9]for the need of an automated system for quick diagnosis, pneumonia detection. Wang et al. [57] proposed a cus- because the manual methods of X-ray analysis are time tomized VGG16 model for lung regions identification to consuming and unable to serve the purpose due to limited classify different types of pneumonia. Later, Ronneburger availability of X-ray machine operators or radiologists. et al. [42] demonstrated the effectiveness of image aug- Amid the COVID-19 outbreak, many companies at the mentation with CNN in the presence of a small set of global level around the world embraced a flurry of Artificial images. In the area of biomedical image classification, Intelligence (AI) based solutions to detect COVID-19 on Rajpurkar et al. [40] proposed a dense CNN with 121-layers chest X-ray scans. It is evident that deep learning tools to detect several pathologies including pneumonia using are effectively used to screen mild cases, triage new chest X-rays. Lakhani et al. [26] obtained an area under the infections, and monitor disease advancements. This way of curve (AUC) of 0.95 in pneumonia detection using AlexNet diagnosis can reduce the growing burden on radiologists, and GoogLeNet along with image augmentation. Recently, and also supplant standard nucleic acid tests as the primary Automated diagnosis of COVID-19... 2691 diagnostic tool for coronavirus infection. It is also reported is used to measure accuracy, false positive rate, F1 score, that a swab test needs isolation for testing procedure, Matthew’s correlation coefficient (MCC) and kappa. It whereas chest X-ray based detection can be easily is found that ResNet50 in combination with SVM is manageable. Kermany et al. [22] proposed CXR image- statistically superior when compared to other models. Later, based deep learning model to detect pneumonia and classify Bukhari et al. [4] also used ResNet-50 CNN architectures other diseases using different medical datasets with testing on 278 CXR images, partitioned under 3 groups as accuracy of 92.80%. In another similar research, Stephen normal, pneumonia and COVID-19. This approach gave et al. [51] illustrated an efficient deep learning approach promising results and indicated substantial differentiation for pneumonia classification, by using four convolutional of pulmonary changes caused by COVID-19 from the other layers and two dense layers in addition to classical image types of pneumonia. augmentation and achieved 93.73% testing accuracy. Later, Recently, in another research work, an improved ResNet- Saraiva et al. [44] experimented convolutional neural 50 CNN architecture named COVIDResNet has been pro- networks to classify images of childhood pneumonia by posed [10], where conventional ResNet-50 model is applied using a deep learning model with seven convolutional layers with different training techniques including progressive along with three dense layers while achieving 95.30% resizing, cyclical learning rate finding, and discriminative testing accuracy. Liang and Zheng [27] demonstrated a learning rates to gain fast and accurate training. The exper- transfer learning method with a deep residual network for iment is performed through progressively re-sizing of input pediatric pneumonia diagnosis with 49 convolutional layers images to 128 × 128 × 3, 224 × 224 × 3 and 229 × 229 × 3 and two dense layers and achieved 96.70% testing accuracy. pixels, and automatic learning rate selection for fine-tuning In similar research, Wu et al. [60] focused on convolutional the network at each stage. This work claimed to be com- deep neural learning networks and random forest to propose putationally efficient and highly accurate for multi-class a pneumonia prediction using CXR images and achieved classification. 97% testing accuracy. A new deep anomaly detection model is developed by Afterwards, Narin et al. [31] proposed a deep convo- Zhang et. al. [62] for fast and more reliable screening. lutional neural network based automatic prediction model To evaluate the model performance, CXR image data of of COVID-19 with the help of pre-trained transfer models COVID-19 cases and other pneumonia has been collected using CXR images. In this research, authors used ResNet50, from two different sources. To eliminate the data imbalance InceptionV3 and Inception-ResNetV2 pre-trained models problem in the collected samples, authors proposed a to obtain a higher prediction accuracy for a subset of CXR based COVID-19 screening model through anomaly X-ray dataset. Apostolopoulos et. al. [3] in their study, detection task [35]. utilised state-of-the-art convolutional neural network archi- Following this context, this article proposes to contribute tectures for classifying the CXR images. Transfer Learning for early diagnosis of COVID-19 using the state-of-the-art was adopted to handle various abnormalities present in deep learning architectures, assisted with transfer learning the dataset. Two datasets from different repositories have and class imbalance learning approaches. been used to study images of three classes: COVID-19, bacterial/viral pneumonia and normal condition. The arti- cle establishes the suitability of the deep learning model 3 Dataset description with the help of accuracy, sensitivity, and specificity parameters. In this research three datasets are utilized for experiments: In another research, generative adversarial networks COVID-19 image [8], Radiological Society of North (GAN) are used by Khalifa et al. [23] to detect pneumonia America (RSNA) [50] and U.S. national library of from CXR images. The authors addressed the overfitting medicine (USNLM) collected Montgomery country - problem and claimed its robustness by generating more NLM(MC) [19]. COVID-19 image dataset is a public images through GAN. The dataset containing 5863 CXR database of pneumonia cases with CXR images related to images of two categories: normal and pneumonia, has been COVID-19, MERS, SARS, and ARDS collected by Cohen used with typical deep learning models such as AlexNet, et al. [8] from multiple resources available at public domains GoogLeNet, Squeeznet and Resnet18 to detect pneumonia. without infringing patient’s confidentiality (Fig. 1b). It is This research highlights that the Resnet18 outperformed claimed that this dataset can help to identify characteristics among other deep transfer models in combination with of COVID-19 in contrast to other types of pneumonia; GAN. Further, Sethy et al. [46] proposed a deep learning therefore it can play a major role in predicting survival rate. based model to identify coronavirus infections using CXR The dataset includes the statistics up to March 25, 2020 images. Deep features from CXR images have been consisting of 5 types of pneumonia such as SARSr-CoV-2 extracted and support vect or machine (SVM) classifier or COVID-19, SARSr-CoV-1 or SARS, Streptococcus spp., 2692 N. S. Punn and S. Agarwal Fig. 1 Sample chest radiographs Pneumocystis spp. and ARDS with following attributes: 4 Proposed contribution patient ID, offset, sex, age, finding, survival, view, modality, date, location, filename, doi, url, license, clinical notes, and The era of artificial intelligence has brought significant other notes. improvements in the living society [30]. The recent Another dataset utilized in this study is published under advancements in deep learning have extended its domain RSNA pneumonia detection challenge is a subset of 30,000 in various applications such as healthcare, pixel restoration, examinations taken from the NIH CXR14 dataset [50]. visual recognition, signal processing, and a lot more [28]. Out of 30,000 selected images, 15,000 examinations had In healthcare domain, the deep learning based image positive cases of pneumonia and from the remaining 15000 processing approaches for classification and segmentation cases, 7500 cases had no findings and other 7500 cases are applied for faster, efficient, and early diagnosis of the had symptoms other than pneumonia. All these images are deadly diseases e.g. breast cancer, brain tumor, etc. by using annotated by a group of experts including radiologists in two different imaging modalities such as X-ray, CT, MRI, [47] stages. A sample image is shown in Fig. 1c. This dataset and fused modalities [37] along with its future possibilities. has been published in two stages. In Stage one, 25,684 The success of these approaches is dependent on the large training images were considered to test 1,000 images. amount of data availability, which however is not in the case Later in stage two 1000 testing samples were added to the of automated COVID-19 detection. training set to form the dataset of 26,684 training images The main contribution of the work is divided into and a new set of 3,000 radiographs were introduced for the components as shown in Fig. 2. It has two concrete the test. For robust testing and comprehensive coverage components: data preprocessing and classification. COVID- of the comparative analysis, NLM(MC) [19] dataset is 19 image, RSNA and NLM(MC) datasets are used to also utilized that consists of 138 chest posterior-anterior x- generate the final working set. The newly generated rays samples of tuberculosis and normal cases. A sample dataset contains CXR images of the following classes: image is represented in Fig. 1d. Table 1 presents the class coronavirus caused diseases, pneumonia, other diseases summary details of the fused dataset resulting from the and normal cases. Further, binary classification (COVID- above discussed datasets which is utilized for training, 19 vs others) and multi-class classification (COVID- testing and validation of the proposed approach. The fused 19, other types of pneumonia, tuberculosis and normal) dataset is composed of 1214 posteroanterior chest x-ray are achieved using random oversampling and weighted samples with classes labeled as COVID-19 (108), other class loss function approaches for unbiased fine-tuned pneumonia (515), tuberculosis (58) and normal (533). The learning (transfer learning) in various state-of-the-art deep generated fused dataset is publicly available [36]. learning approaches such as baseline ResNet, Inception-v3, Inception ResNet-v2, DenseNet169, and NASNetLarge [17, 52, 54, 63]. The trained models are utilized for identification and classification of COVID-19 in novel samples. Later, Table 1 Fused dataset disease class summary details visualization techniques are utilized to understand and Dataset Findings PCXR images Total samples elaborate the basis of the classification results. COVID-19 COVID-19 108 153 4.1 Imbalanced learning approach Other pneumonia 45 RSNA Normal 453 923 Class balancing techniques are necessary when minority Other pneumonia 470 classes are more important. The dataset used in this research NLM(MC) Tuberculosis 58 138 is highly imbalanced which may lead to biased learning of Normal 80 the model. Number of coronavirus infected CXR images are Automated diagnosis of COVID-19... 2693 Fig. 2 Schematic representation of the proposed components for COVID-19 identification very less compared to other classes, hence class balancing statistical operations. In this work, the samples of CXR image of COVID-19 positive cases are less as compared techniques must be insured to smoothen the learning to other classes, therefore, these minority class images are process. This section discusses two approaches to handle randomly oversampled by means of rotation, scaling, and the class imbalance problem: weight class approach and displacement with the objective to achieve equal distribution random oversampling [21]. of classes and accommodate unbiased learning among the deep learning models. 4.1.1 Weighted class approach 4.2 Classification Strategy In this approach, the intention is to balance the data by altering the weights that each training sample class carries Based on the type of data samples availability of CXR when computing the loss. Normally, each class carries images the COVID-19 classification is divided into two equal weights, but sometimes certain classes with minority following schemes: samples are required to hold more weights if they are more important because training examples within that class – Binary Classification - In this classification scheme, the should have a significant effect on the loss function. In the coronavirus positive samples labelled as “1” (COVID- used dataset the coronavirus infected image class samples 19) are identified against the rest of the samples labelled must be given more weights as they are more significant. In as “0” (non COVID-19 case) which involves other cases this article, the weights for each class is generated based on e.g. chlamydophila, SARS, streptococcus, tuberculosis, the Eq. (1). etc., along with the normal cases. – Multi-class Classification—In this classification scheme, the aim is to distinguish and identify the c=0 w(c) = C . (1) COVID-19 samples from the other pneumonia cases N .n along with the presence of tuberculosis and normal case where C is the class constant for a class c, N is the number c findings. The multi-class classification is performed of classes, and n is the number of samples in a class c. The with three and four classes. The three classes are pro- computed class weights are later fused with the objective vided with labels as “0” being a normal case, “1” being function (loss function) of the deep learning model in order a COVID-19 case, and “2” being other pneumonia and to heavily penalize the false predictions concerned with the tuberculosis cases, whereas four classes are labeled minority samples, which in this case is coronavirus. as “0” being a normal case, “1” being a COVID-19 case, and “2” being other pneumonia case and “3” as 4.1.2 Random oversampling approach tuberculosis case. In both the classification strategies, the deep learning In this approach, the objective is to increase the number models are trained with the above discussed imbalanced of minority samples by utilizing the existing samples learning approaches using the weighted categorical cross belonging to the minority class. The minority samples entropy (WCE) loss function as given by Eq. (2)and are increased until the samples associated with every Eq. 3 [24]: class become equal. Hence the procedure follows by identifying the difference between the number of samples in majority and minority class. To fill this void of difference, f(s) = (2) the samples are generated from the randomly selected sample belonging to the minority class by applying certain C 2694 N. S. Punn and S. Agarwal where i(x,y) is an input image, max and min are max th th WCE =− w(i).t .log(f (s) ) (3) i i and min thresholds to design the mask. Despite filtering the unwanted information, there is still the possibility of In categorical cross entropy, the distribution of the uncertainty at the deep pixel level representation [15]. The predictions (the activations in the output layer, one for each denoising or removal of such uncertainty is carried through class) is compared with the true distribution only, to ensure the adaptive total variation method [53] while preserving the the clear representation of the true class as one-hot encoded original distribution of pixel values. vector; here, closer the model’s outputs are to that vector, Let for a given grayscale image f, on a bounded set  over 2 2 the lower the loss. R ,where  ⊂ R , denoising image u that closely matches to observed image x = (x ,x )  -pixels, givenas 1 2 4.3 Data Preprocessing u = arg min (u − f .ln u)dx + (ω(x)| u|dx) In this article, due to the limited samples of posteroanterior chest X-ray images concerned with positive COVID- (5) 19 [8] cases, the data samples are mixed with the other randomly selected CXR images selected from other where ω(x) = , G - the Gaussian kernel 1+k mod G ∗u datasets-, RSNA [50] and NLM(MC) [19]. The RSNA for smoothing with σ variance, k > 0 is contrast parameter and NLM(MC) datasets consists of posteroanterior CXR and * is convolution operator. images covering sample cases labelled as pneumonia Figure 3 illustrates the data preprocessing stages by and tuberculosis respectively along with normal samples. considering an instance of COVID-19 case consisting of Table 2 describes the distribution of training, testing, and textual and symbolic artifacts from the generated dataset. validation sets using the fused dataset for binary and multi- The resulting distributed pixels histograms at each stage class classification along with different class imbalance of preprocessing shown in Fig. 3, illustrates that the strategies i.e. class weighted loss function that penalizes preprocessing approach tends to preserve the original nature the model for any false negative prediction and random of distribution of the pixels while removing the irregular oversampling [61] of minority classes which in this case is intensities. The preprocessed images are then divided COVID-19. into training, testing, and validation set for training and The CXR images in the aggregated dataset also consists evaluation of the state-of-the-art deep learning classification of unwanted artifacts such as bright texts, symbols, varying models. resolutions and pixel level noise, which necessitates its preprocessing. In order to suppress the highlighted textual 4.4 Deep learning models and symbolic noise, the images are inpainted with the image mask generated using binary thresholding [34]asgiven by This section incorporates the state-of-the-art deep learning Eq. (4), followed by resizing the images to a fixed size models utilized in the present research work as shown resolution of 331 × 331 × 3. in Table 3 along with their respective contribution, max th, i(x, y) ≥ min th. parameters, and performance on the standard benchmark M(x, y) = (4) 0, otherwise. datasets. The inception deep convolutional architectures Table 2 Posteroanterior CXR images distribution into training, validation, and test sets from the fused datasets for different problem definitions Findings Class imbalanced learning strategy Classification with class weighted loss function Classification with random oversampling Binary Multi-class Multi-class Binary Multi-class Multi-class (2 classes) (3 classes) (4 classes) (2 classes) (3 classes) (4 classes) Tr Val Tst Tr Val Tst Tr Val Tst Tr Val Tst Tr Val Tst Tr Val Tst Normal 906 90 110 437 44 52 437 44 52 960 90 110 469 44 52 437 44 52 Tuberculosis 469 46 58 47 5 6 469 46 58 437 5 6 Other pneumonia 422 41 52 437 41 52 COVID-19 88 9 11 88 9 11 88 9 11 960 9 11 469 9 11 437 9 11 Total 994 99 121 994 99 121 994 99 121 1920 99 121 1407 99 121 1748 99 121 Automated diagnosis of COVID-19... 2695 Fig. 3 Data preprocessing stages of raw posteroanterior CXR image proposed by GoogLeNet are considered as the state-of- Later, Inception-ResNet-v2 was proposed by Szegedy et al. the-art deep learning architectures for image analysis and [52]. This hybrid model is a combination of residual object identification with the basic model as inception- connections and a recent version of Inception architecture. v1 [5]. Later, this base model was refined by introducing It is intended to train very deep convolutional models by the batch normalization and established as the inception- the additive merging of signals, both for image recognition v2 [54]. In further iterations, additional factorisation was and object detection. This network is more robust and learns introduced and released as the inception-v3. It is one of the rich feature representations. Afterwards, DenseNet was pre-trained models to perform two types of specific tasks: proposed by Huang et al. [17]. It works on the concept of dimensionality reduction using CNN and classification reuse, in which each layer receives inputs from all previous using fully-connected and softmax layers. Since it is layers and yields a condensed model to pass its own feature- originally trained on over a million images consisting of maps to all subsequent layers. This makes the network 1,000 classes of ImageNet, its head layers can be retrained thinner and compact with the fewer number of channels, for the generated dataset using historical knowledge to while improving variation in the input of subsequent layers, reduce the extensive training and computational power. and becomes easy to train and highly parameter efficient. Table 3 Recent deep learning Architecture Year Contribution Param. Dataset E. rate architectures that are reported with top 5 error rate per year Inception-v3 2015 Avoided the bottleneck 23.6M ILSVRC 5.6 [54] representations, Dimension reduction promotes faster learning. Inception 2016 Focused on residual 56M ILSVRC 4.9 ResNet-v2 connection rather [52] than filter connection approach. DenseNet169 2017 Dense connection blocks 17.8M CIFAR-10 5.19 [17] for activation flow across CIFAR-100 19.64 the layers. CIFAR-10+ 3.46 CIFAR-100+ 17.18 NASNetLarge 2018 Search and transfer the 22.6M CIFAR-10 2.4 [63] architecture block from small dataset to large dataset, Scheduled drop path regularization. 2696 N. S. Punn and S. Agarwal Google Brain research team proposed NASNet model based its trainable parameters along with the addition of fully on reinforcement learning search methods [63]. It creates connected layers with 128 neurons and 2 or 3 neurons search space by factoring the network into cells and further in the output layer depending on binary or multi-class dividing it into a number of blocks. Each block is supported classification, accompanied with the softmax activation by a set of popular operations in CNN models with various function. kernel size e.g: convolutions, max pooling, average pooling, dilated convolution, depth-wise separable convolutions etc. 5 Evaluation metrics 4.5 Fine tuning The deep learning models are trained using the train- Training deep learning models require (Inception-v3, ing and validation set under consideration of each pos- InceptionResNet-v2, etc.) exhaustive amount of resources sible combination of the discussed approaches of class and time. While these networks, attain relatively excellent imbalance learning and classification strategy, thereby mak- performance on ImageNet [9], training them on a CPU is ing four possible scenarios for training a model. Later, an exercise in futility. These CNNs are often trained for the trained models are evaluated on the test set using a couple of weeks or more using arrays of GPUs to get the standard benchmark performance metrics such as good results on the complex and complicated datasets. In accuracy, precision, recall (selectivity), area under curve most deep CNNs the first few convolution layers learn (AUC), specificity and F1 score (dice coefficient) as shown low-level features (edges, curves, blobs) and with progress, in Fig. 5. through the network, it learns more mid/high-level features or patterns associated with the on-going task. In fine tuning, the aim is to keep or freeze these trained low-level features, 6 Training and testing and only train the high-level features needed for our new image classification problem. The proposed approach is trained and tested on the above In this article, the trials of training the deep learning discussed fused dataset using mini-batch gradient descent classification models initiates with the baseline residual with learning rate optimizer as adam [43]via N-Vidia network that is composed of five residual blocks [16], Titan GPU. The training, testing, and validation sets are defined by two convolutions whose rectified linear unit highlighted in Table 2. The training phase is assisted (ReLU) activation is concatenated with the input of the first with the batch size of 10, 4-fold cross validation, switch convolution layer, followed by max-pooling and instance normalization [29] to adapt to instance or batch or layer normalization, except the final output layer which uses normalization, and earlystopping technique that aims to stop softmax activation function. Later, transfer learning is the training process if the validation error stops decreasing. utilized on the state-of-the-art architectures discussed in For each epoch the above discussed evaluation metrics are Table 3 to utilize the pre-trained models while fine tuning computed on the training and validation set to analyse the the head layers to serve the purpose of classifying the model’s performance and improve the classification results. COVID-19 samples, as illustrated in Fig. 4. This follows Later, the test set is utilized to evaluate the results of the by enabling the four head layers of the network to adjust proposed approach. Fig. 4 Schematic representation of training framework for deep learning architectures via transfer learning Automated diagnosis of COVID-19... 2697 Fig. 5 Confusion matrix and performance evaluation metrics 7 Results and discussion the training phase for classification of COVID-19 samples, generated using the tensorboard. Since the models are s With extensive trials, it is observed that there is no initialized with the trained weights on the ImageNet and individual model that displayed best performance for all the head layers are fine-tuned for classification, the training of scenarios in terms of accuracy, precision, recall, specificity, the models initiates with descent metrics values. The best AUC, and F1-score. However, the NASNetLarge mostly scores of the models corresponding to each scenario are claimed the best scores, thus making it best fit for the highlighted in Table 4 along with the detailed comparative classification of COVID-19 samples. It is also observed analysis of the results for classifying the COVID-19 samples that the results of binary classification (COVID-19 vs non- in terms of discussed evaluation metrics for different COVID-19) are better than the multi-class classification scenarios CB, CM ,CM RB, RM and RM where C and 3 4 3 4 (COVID-19 vs other classes). With this it is evident that R indicates class imbalanced learning approaches, namely by grouping other diseases together as non-COVID-19 weighted class loss functions and random oversampling samples, models can efficiently learn features and patterns of the minority class, whereas B indicates the binary belonging to the COVID-19. Figure 6 shows the average classification and M (3 classes), M (4 classes) indicate 3 4 performance curves of the NASNetLarge model evaluated multi-class classification schemes. Furthermore, the results and monitored on the test set for each iteration during obtained are compared with the other recently proposed Fig. 6 Monitoring every epoch of training with performance curves for the NASNetLarge model with average a accuracy, b AUC, c specificity, d F1-score, e precision, and f recall. The curves are smoothed using moving exponential average 2698 N. S. Punn and S. Agarwal Table 4 Comparative analysis Model Scenario Accuracy Precision Recall AUC Specifcity F1-score of the classification results of the deep learning models for Baseline ResNet CB 0.84 0.67 0.89 0.90 0.85 0.76 each scenario CM 0.77 0.62 0.89 0.90 0.77 0.73 CM 0.77 0.61 0.83 0.90 0.77 0.70 RB 0.89 0.56 0.56 0.89 0.60 0.56 RM 0.76 0.63 0.42 0.83 0.77 0.50 RM 0.76 0.65 0.43 0.82 0.73 0.52 Inception v2 CB 0.88 0.77 0.79 0.93 0.89 0.78 CM 0.87 0.73 0.88 0.90 0.88 0.80 CM 0.87 0.73 0.85 0.89 0.85 0.79 RB 0.88 0.78 0.70 0.88 0.91 0.74 RM 0.87 0.90 0.81 0.91 0.90 0.85 RM 0.84 0.89 0.81 0.91 0.88 0.85 Inception ResNet v2 CB 0.95 0.86 0.66 0.98 0.87 0.75 CM 0.92 0.85 0.92 0.99 0.89 0.88 CM 0.91 0.83 0.91 0.98 0.89 0.87 RB 0.90 0.85 0.75 0.93 0.88 0.80 RM 0.92 0.97 0.96 0.99 0.94 0.96 RM 0.91 0.95 0.93 0.98 0.93 0.94 DenseNet 169 CB 0.90 0.82 0.78 0.91 0.91 0.80 CM 0.87 0.87 0.91 0.95 0.89 0.89 CM 0.86 0.87 0.89 0.93 0.89 0.88 RB 0.95 0.94 0.96 0.97 0.95 0.95 RM 0.91 0.93 0.95 0.93 0.95 0.94 RM 0.92 0.94 0.95 0.94 0.95 0.94 NASNet Large CB 0.97 0.82 0.91 0.99 0.98 0.86 CM 0.94 0.89 0.91 0.94 0.89 0.90 CM 0.95 0.88 0.89 0.92 0.89 0.88 RB 0.98 0.87 0.90 0.99 0.94 0.98 RM 0.96 0.93 0.91 0.96 0.94 0.92 RM 0.95 0.95 0.90 0.95 0.94 0.92 Bold numbers highlight the best scores of the models corresponding to each scenario approaches to detect COVID-19 positive symptoms via X- proposed approach outperforms other approaches, whereas ray images as shown in Table 5. It is observed that the Apostolopoulos et al. [3] achieved similar results with the Table 5 Comparative analysis of the proposed approach with recently proposed strategies Authors Technique Network Classes Accuracy Precision Recall AUC Specificity Yujin et al. [33] Fine tuning Res- Net18 4 0.88 0.83 0.86 – 0.96 Afshar et al. [1] Fine tuning Capsule 0.95 – 0.90 0.95 Proposed Fine NASNet Large 0.95 0.95 0.90 0.94 0.92 Wang et al. [55] Full training COVID-Net 3 0.92 0.91 0.88 – – Apostolopoulos et al. [3] Fine tuning VGG19 0.87 – 0.92 – 0.98 Proposed Fine tuning NASNet Large 0.96 0.93 0.91 0.96 0.94 Hall et al. [14] Fine tuning Ensemble 2 0.91 – 0.78 0.94 0.93 Apostolopoulos et al. [3] Fine tuning VGG19 0.98 – 0.92 – 0.98 Proposed Fine tuning NASNet Large 0.98 0.87 0.90 0.99 0.98 Bold numbers highlight the results obtained using proposed approach Automated diagnosis of COVID-19... 2699 Fig. 7 LIME explanation of three distinct class samples using NASNetLarge along with the prediction probabilities of sample being normal (N), COVID-19 (C), and other pneumonia (OP) help of VGG19 model [49] but with approximately double There are many visualization techniques for example class trainable parameters (144M) as compared to the proposed activation maps (CAM) [45], saliency maps (SM) [48], local approach (83M). interpretable model-agnostic explanations (LIME) [41], and a lot more. In this article, activation maps and LIME techniques are utilized to present the model perception of 8 Visualization identifying and classifying the COVID-19 samples from CXR images. CAM aims at understanding the feature space Training the model and getting the results is not sufficient of an input image that influences the prediction, whereas unless it is understood that what is triggering the concerned LIME is an innovative explanation technique to represent output. To handle this blackbox, visualization techniques the model prediction with local fidelity, interpretability assist in illustrating the basis of prediction of the model. and model agnostic. For instance, fine-tuned NASNetLarge Fig. 8 CAM of NASNetLarge model for a Normal, b COVID-19, and c Other pneumonia 2700 N. S. Punn and S. Agarwal architecture is considered to generate LIME explanations convolutional neural networks. Phys Eng Sci Med 1:635–640. https://doi.org/10.1007/s13246-020-00865-4 for some samples taken from the test set (see Fig. 7), 4. Bukhari SUK, Bukhari SSK, Syed A, SHAH SSH (2020) The whereas class activation maps tend to present the patterns diagnostic evaluation of convolutional neural network (cnn) for learned by the model for classification of samples as shown the assessment of chest x-ray of patients infected with covid-19. in Fig. 8. Figure 7 presents the LIME technique applied to medRxiv 5. Chen Q, Montesinos P, Sun QS, Heng PA et al (2010) Adaptive four samples belonging to four distinct classes as COVID- total variation denoising based on difference curvature. Image Vis 19, other types of pneumonia, tuberculosis and normal Comput 28(3):298–306 cases. The red and green areas in the LIME generated expla- 6. Choe J, Lee SM, Do KH, Lee G, Lee JG, Lee SM, Seo JB nation correspond to the regions that contributed against the (2019) Deep learning–based image conversion of ct reconstruction kernels improves radiomics reproducibility for pulmonary nodules predicted class and towards the predicted class respectively. or masses. Radiology 292(2):365–373 7. Chouhan V, Singh SK, Khamparia A, Gupta D, Tiwari P, Moreira C, Damase ˇ vicius ˇ R, de Albuquerque VHC (2020) A novel transfer 9 Conclusion learning based approach for pneumonia detection in chest x-ray images. Appl Sci 10(2):559 8. Cohen JP, Morrison P, Dao L (2020) Covid-19 image data This article proposes to leverage the state-of-the-art deep collection. arXiv:2003.11597 learning models to aid in early identification and diagnosis 9. Deng J, Dong W, Socher R, Li L, Kai L, Li F-F (2009) Imagenet: of COVID-19 virus by using the limited posteroanterior a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248– chest X-ray images. Each trained model was evaluated using benchmark performance metrics e.g. accuracy, precision, 10. Farooq M, Hafeez A (2020) Covid-resnet: a deep learn- recall, area under curve, specificity, and F1 score under four ing framework for screening of covid19 from radiographs. different scenarios concerned with imbalanced learning and arXiv:2003.14395 11. Fourcade A, Khonsari R (2019) Deep learning in medical image classification strategy. With extensive trials, it was observed analysis: a third eye for doctors. J Stomatol Oral Maxillofac Surg that models achieve different scores in different scenarios, 120(4):279–288 among which NASNetLarge displayed better performance 12. Gomez ´ P, Semmler M, Schutzenber ¨ ger A, Bohr C, Dollinger ¨ M specially in binary classification of COVID-19 samples. (2019) Low-light image enhancement of high-speed endoscopic videos using a convolutional neural network. Med Biol Eng The visual representation based on local interpretable model Comput 57(7):1451–1463 agnostic explanations is utilized to understand the basis of 13. Guo YR, Cao QD, Hong ZS, Tan YY, Chen SD, Jin HJ, Tan KS, prediction of the model. As an extension to this work more Wang DY, Yan Y (2020) The origin, transmission and clinical deep learning models and preprocessing techniques can be therapies on coronavirus disease 2019 (covid-19) outbreak–an update on the status. Mil Med Res 7(1):1–10 explored to achieve better results. 14. Hall LO, Paul R, Goldgof DB, Goldgof GM (2020) Finding covid- Acknowledgements We are very much obliged to our institute, Indian 19 from chest x-rays using deep learning on a small dataset. Institute of Information Technology Allahabad (IIITA), India and Big arXiv:2004.02060 Data Analytics (BDA) lab for ensuring the needed resources and 15. Hasinoff SW (2014) Photon, poisson noise. Computer Vision: A support. We also would like to extend our thanks to the colleagues Reference Guide, Boston, pp 608–610 for their valuable guidance and suggestions. The manuscript is also 16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for available in the pre-print arXiv with the eprint-2004.11676 under the image recognition. In: Proceedings of the IEEE conference on license Attribution 4.0 International (CC BY 4.0). computer vision and pattern recognition, pp 770–778 17. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of Compliance with Ethical Standards the IEEE conference on computer vision and pattern recognition, pp 4700–4708 Conflict of interest The authors have no conflict of interest to declare. 18. Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, Marklund H, Haghgoo B, Ball R, Shpanskaya K et al (2019) Ethical approval This article does not contain any studies with human Chexpert: a large chest radiograph dataset with uncertainty labels participants or animals performed by any of the authors. and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 590–597 19. Jaeger S, Candemir S, Antani S, Wang ´ YXJ, Lu PX, Thoma G (2014) Two public chest x-ray datasets for computer-aided References screening of pulmonary diseases. Quant Imaging Med Surg 4(6):475 1. Afshar P, Heidarian S, Naderkhani F, Oikonomou A, Plataniotis 20. Joaquin AS (2020) Using deep learning to detect pneumonia KN, Mohammadi A (2020) Covid-caps: a capsule network-based caused by ncov-19 from x-ray images. https://towardsdatascience. framework for identification of covid-19 cases from x-ray images. com/using-deep-learning-to-detect-ncov-19-from-x-ray-images-1 arXiv:2004.02696 a89701d1acd (Online; Accessed 12 April 2020) 2. Allen JN, Davis WB (1994) Eosinophilic lung diseases. Am J 21. Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning Respir Crit Care Med 150(5):1423–1438 with class imbalance. J Big Data 6(1):27 3. Apostolopoulos ID, Mpesiana TA (2020) Covid-19: automatic 22. Kermany DS, Goldbaum M, Cai W, Valentim CC, Liang H, Baxter detection from x-ray images utilizing transfer learning with SL, McKeown A, Yang G, Wu X, Yan F et al (2018) Identifying Automated diagnosis of COVID-19... 2701 medical diagnoses and treatable diseases by image-based deep 42. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional learning. Cell 172(5):1122–1131 networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted 23. Khalifa NEM, Taha MHN, Hassanien AE, Elghamrawy S (2020) intervention. Springer, pp 234–241 Detection of coronavirus (covid-19) associated pneumonia based 43. Ruder S (2016) An overview of gradient descent optimization on generative adversarial networks and a fine-tuned deep transfer algorithms. arXiv:1609.04747 learning model using chest x-ray dataset. arXiv:2004.01184 44. Saraiva A, Ferreira N, Sousa L, Carvalho da Costa N, Sousa 24. Koidl K (2013) Loss functions in classification tasks. School of J, Santos D, Soares S (2019) Classification of images of Computer Science and Statistic Trinity College, Dublin childhood pneumonia using convolutional neural networks. In: 6th 25. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classifi- International conference on bioimaging, pp 112–119 cation with deep convolutional neural networks. In: Advances in 45. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra neural information processing systems, pp 1097–1105 D (2017) Grad-cam: visual explanations from deep networks 26. Lakhani P, Sundaram B (2017) Deep learning at chest radiogra- via gradient-based localization. In: Proceedings of the IEEE phy: automated classification of pulmonary tuberculosis by using international conference on computer vision, pp 618–626 convolutional neural networks. Radiology 284(2):574–582 46. Sethy PK, Behera SK (2020) Detection of coronavirus disease 27. Liang G, Zheng L (2019) A transfer learning method with deep (covid-19) based on deep features. Preprints, 2020030300, 2020 residual network for pediatric pneumonia diagnosis. Computer 47. Shen D, Wu G, Suk HI (2017) Deep learning in medical image Methods and Programs in Biomedicine, pp 104964 analysis. Ann Rev Biomed Eng 19:221–248 28. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A 48. Simonyan K, Vedaldi A, Zisserman A (2013) Deep inside survey of deep neural network architectures and their applications. convolutional networks: visualising image classification models Neurocomputing 234:11–26 and saliency maps. arXiv:1312.6034 29. Luo P, Ren J, Peng Z, Zhang R, Li J (2018) Differ- 49. Simonyan K, Zisserman A (2014) Very deep convolutional entiable learning-to-normalize via switchable normalization. networks for large-scale image recognition. arXiv:1409.1556 arXiv:1806.10779 50. Stein A (2020) Pneumonia dataset annotation methods. rsna pneu- 30. Makridakis S (2017) The forthcoming artificial intelligence (ai) monia detection challenge discussion, 2018. https://www.kaggle. revolution: its impact on society and firms. Futures 90:46–60 com/c/rsna-pneumonia-detection-challenge/discussion/ (Online; 31. Narin A, Kaya C, Pamuk Z (2020) Automatic detection of Accessed 12 April 2020) coronavirus disease (covid-19) using x-ray images and deep 51. Stephen O, Sain M, Maduh UJ, Jeong DU (2019) An efficient convolutional neural networks. arXiv:2003.10849 deep learning approach to pneumonia classification in healthcare. 32. Oakden-Rayner L (2020) Chexnet: an in-depth review. J Healthcare Eng 2019. https://doi.org/10.1155/2019/4180949 https://lukeoakdenrayner.wordpress.com/2018/01/24/ 52. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception- chexnet-an-in-depth-review/ (Online; Accessed 12 April 2020) v4, inception-resnet and the impact of residual connections 33. Oh Y, Park S, Ye JC (2020) Deep learning covid-19 features on on learning. In: Thirty-first AAAI conference on artificial cxr using limited training data sets. IEEE Trans Med Imaging intelligence 39:2688–2700. https://doi.org/10.1109/TMI.2020.2993291 53. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, 34. OpenCV (2020) Image thresholding. https://docs.opencv.org/ Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with master/d7/d4d/tutorial py thresholding.html (Online; Accessed convolutions. In: Proceedings of the IEEE conference on computer 12 April 2020) vision and pattern recognition, pp 1–9 35. Pang G, Shen C, van den Hengel A (2019) Deep anomaly 54. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) detection with deviation networks. In: Proceedings of the 25th Rethinking the inception architecture for computer vision. In: ACM SIGKDD international conference on knowledge discovery Proceedings of the IEEE conference on computer vision and & data mining, pp 353–362 pattern recognition, pp 2818–2826 36. Punn N (2020) Covid-19 posteroanterior chest x-ray fused 55. Wang L, Wong A (2020) Covid-net: a tailored deep convolutional (cpcxr) dataset. https://github.com/nspunn1993/COVID-19-PA- neural network design for detection of covid-19 cases from chest CXR-fused-dataset (Online; Accessed 21 Jul 2020) radiography images. arXiv:2003.09871 37. Punn NS, Agarwal S (2020) Inception u-net architecture for 56. Wang S, Kang B, Ma J, Zeng X, Xiao M, Guo J, Cai M, Yang J, Li semantic segmentation to identify nuclei in microscopy cell Y, Meng X et al (2020) A deep learning algorithm using ct images images. ACM Trans Multimed Comput Commun Appl (TOMM) to screen for corona virus disease (covid-19). MedRxiv 16(1):1–15 57. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers R (2017) 38. Punn NS, Sonbhadra SK, Agarwal S (2020) Covid-19 epidemic Hospital-scale chest x-ray database and benchmarks on weakly- analysis using machine learning and deep learning algorithms supervised classification and localization of common thorax medRxiv diseases. In: IEEE CVPR 39. Rajinikanth V, Dey N, Raj ANJ, Hassanien AE, Santosh K, 58. WHO (2020) Coronavirus disease 2019 (covid-19) situ- Raja N (2020) Harmony-search and otsu based system for ation report–81. https://www.who.int/docs/default-source/ coronavirus disease (covid-19) detection using lung ct scan coronaviruse/situation-reports/. Online; Accessed 12 April 2020 images. arXiv:2004.03431 59. WHO (2020) Who timeline—covid-19. https://www.who.int/ 40. Rajpurkar P, Irvin J, Ball RL, Zhu K, Yang B, Mehta H, Duan news-room/detail/08-04-2020-who-timeline---covid-19 (Online; T, Ding D, Bagul A, Langlotz CP et al (2018) Deep learning Accessed 12 April 2020 for chest radiograph diagnosis: a retrospective comparison of the 60. Wu H, Xie P, Zhang H, Li D, Cheng M (2020) Predict pneumonia chexnext algorithm to practicing radiologists. PLoS Med 15(11): with chest x-ray images based on convolutional deep neural e1002686 learning networks. J Intell Fuzzy Syst (Preprint) 1–15 41. Ribeiro MT, Singh S, Guestrin C (2016) “Why should I 61. Yap BW, Rani KA, Rahman HAA, Fong S, Khairudin Z, Abdullah trust you?” Explaining the predictions of any classifier. In: NN (2014) An application of oversampling, undersampling, Proceedings of the 22nd ACM SIGKDD international conference bagging and boosting in handling imbalanced datasets. In: on knowledge discovery and data mining, pp 1135–1144 2702 N. S. Punn and S. Agarwal Proceedings of the first international conference on advanced data Dr. Sonali Agarwal is and information engineering (DaEng-2013). Springer, pp 13–22 presently working as Asso- 62. Zhang J, Xie Y, Li Y, Shen C, Xia Y (2020) Covid-19 screening on ciate Professor in department chest x-ray images using deep learning based anomaly detection. of information technology arXiv:2003.12338 at IIIT Allahabad, Prayagraj, 63. Zoph B, Vasudevan V, Shlens J, Le QV (2018) Learning U. P., India 211015. She is transferable architectures for scalable image recognition. In: having hands on experience Proceedings of the IEEE conference on computer vision and on stream computing and pattern recognition, pp 8697–8710 complex processing plat- forms such as Apache Spark, Publisher’s note Springer Nature remains neutral with regard to Apache Flink and ESPER. jurisdictional claims in published maps and institutional affiliations. Her research interests are in the areas of Stream Analyt- ics, Big Data, Stream Data Mining, Complex Event Narinder Singh Punn is Processing System, Support presently pursuing his PhD Vector Machines and Software Engineering. from Indian Institute of Information Technology Allahabad, Prayagraj, U. P., India 211015. He is working as a senior research fellow in the “Big Data Analyt- ics” laboratory of IIITA. His research areas are: Machine Learning, Deep Learning and Biomedical Image Analysis.

Journal

Applied Intelligence – Pubmed Central

Published: Oct 17, 2020

There are no references for this article.