Improved Segmentation of Pulmonary Nodules Using Soft Computing Techniques with SegNet and Adversarial Networks

Abstract

Lung cancer is seen as one of the most common lung diseases. For the patients having symptoms, the presence of lung nodules is checked by using various imaging techniques. Pulmonary nodules are detected in most of the cases having symptoms. But identifying the type of the nodule and the categorization still remains as a challenge. After confirming the presence of a nodule (benign or malignant) it takes several other steps to identify its characteristics. Improved imaging methods produce results within a short span of time. Research works are being conducted to increase the overall efficiency of the system. The proposed system considers authentic data sources for the study. The benign and malignant samples are considered for the generation of realistic large image sets. The generation of a large data set with the help of a generative adversarial network (GAN) is the first part of the work. The generated images using GAN cannot be differentiated from the original images even by a trained radiologist. This proves the importance of images generated using GAN. A GAN is able to generate 1024 × 1024 resolutions for natural images. Real data images are used to finetune the SegNet output. Through transfer learning, these weights are transferred to the system for segmentation of the images. The training process use real and generated images, which improve theefficiency of the network. The original data from LUNA 16 was used to further generate benign and malignant samples using GAN. A total of 440 images and their augmented images were used for training the GAN, and it generated 1,001,000 images. Hence the overall efficiency of the system was improved. To verify the results, the same various combinations and methods were considered and tabulated with various parameters. Methods with SegNet, GAN, and other combinations were evaluated to verify the efficiency of the system. Receiver operating characteristics were also plotted and compared with the area under the curve for verification of the results.

1. Introduction

The characteristic of cancerous cells is their growth in the tissues in an uncontrolled manner. It is a serious condition which is to be treated in no time to avoid further spread. The most common reason for lung cancer is the practice of tobacco smoking. There are also some other reasons, such as passive smoking, air pollution, asbestos etc. With the help of computer tomography and radiography, the presence of lung cancer is identified. Through biopsy, the presence of cancer is confirmed. In order to give proper medication in an effective manner to a cancer patient, it is necessary to identify the disease at an early stage [1]. However, it is very difficult to identify early stage disease from CT images even by very experienced professionals. Thus, it is very essential to find out efficient methods to identify the disease. Various deep learning models are showing promising results, and computer aided detection systems help to make the process simple and fast [2]. But the difficult part here is the collection of data needed for research purposes. The unavailability of reliable data sources and the lack of data sources with proper labels make research in this area very difficult. Here in the proposed paper, the required data was generated with the help of a generative adversarial network. With the help of encoder and decoder networks available in a semantic segmentation model, the lung nodule classification was done efficiently. Similarly, the proposed system used SegNet-based architecture with five convolution layers over the U Net-based existing architecture with three convolution layers. Hence, the proposed system resolves the issue related to the availability of large amounts of data for training purposes. The transfer learning process in the proposed system adds the advantage of information transfer, in the form of weights, to the new network. The transfer learning process is another technique included in the proposed system for making the training and segmentation process easy and more efficient. Thus, the proposed system overcomes the drawbacks of the existing system and proposes an efficient method which involves the use of a GAN for image generation, transfer learning techniques for using the weights of a pre-trained network, and SegNet for the segmentation process [3]. The proposed method of having a GAN along with image augmentation was compared with other methods. GAN with SegNet showed very small deviation from the proposed method. This clearly shows the advantage of using GAN for generating images for nodule segmentation. The method was also compared with other encoder and decoder architectures. Medical imaging is an important tool for the diagnosis and treatment of various diseases. In particular, computed tomography (CT) imaging is widely used for the diagnosis of lung cancer, which is one of the leading causes of cancer-related deaths worldwide. Segmentation of the pulmonary nodules in CT images is a critical step in the diagnosis of lung cancer. However, manual segmentation of pulmonary nodules is a time-consuming and error-prone task, and may not be feasible for large-scale studies. Deep learning techniques have shown promising results in automating the segmentation of pulmonary nodules, but the availability of annotated data is often limited in the medical domain. In this context, the combination of generative adversarial networks (GAN) and SegNet has emerged as a potential solution for generating additional training data and improving the segmentation accuracy of pulmonary nodules in CT images. In this article, we present a study that investigates the performance of a GAN-enhanced SegNet model for the automated detection and segmentation of pulmonary nodules in CT images. The proposed model was evaluated on a publicly available dataset and compared with other deep learning models in terms of accuracy and AUC. The results demonstrate the effectiveness of the proposed model in improving the segmentation accuracy and reducing the need for large amounts of manually annotated data. The study provides valuable insights into the potential of GAN-enhanced SegNet for improving the accuracy and efficiency of lung nodule segmentation in CT images, with potential applications in clinical practice and research. The performance comparison was tabulated. Finally, the receiver operating characteristic (ROC) curve of SegNet while training the model with GAN and without GAN was also investigated. The area under the curve (AUC) for SegNet with GAN was found to be 0.851 while that of SegNet was 0.76. The proposed method stood ahead in all respects considered. The proposed work, with the help of available data sets, managed to generate sufficient amounts of data with the help of a GAN. Perfect samples were considered here for the generation of data. Hence, the data generated assured the quality that was used for training the network. For training the GAN, 440 images and augmented images were used. A total of 1,001,000 images were generated and used for training the SegNet architecture. The segmentation showed improved accuracy in the case of benign samples, and it had a perfect shape compared to malignant samples. Through the evaluation of the parameters, it is clear that the GAN along with image augmentation shows better performance.

2. Review of the Literature

The detection and segmentation of lung images are aggressive research areas where many promising results are received. The research results are very much helpful in determining the medicines and to study their effectiveness [1,4]. Textural analyses of medical images are also very helpful in identifying the tissues abnormalities. Accuracy of the results holds a very important role in the recovery of the patient. The best observations result in prescribing the best medicines.

Segmentation can be done manually, but the chances of obtaining inaccurate results are much higher. Semi-automated segmentation is also available. It is error free when compared with the former method. For fast and accurate results, it is desirable to use an automatic segmentation process [5,6,7,8,9,10,11]. There are so many deep learning experiments and research taking place. A drastic development has occurred in the area of classification segmentation and detection of nodules. Automatic nodule detection is an advanced method that can be used to identify various tissues of abnormalities or nodules. Computer-aided detection (CAD) plays an important role in segmentation and classification problems. Comparative studies and analysis help to identify various aspects of CAD systems that are employed in different scenarios [2,12]. Another CAD system using an algorithm that is free of parameters was proposed to increase the efficiency of nodule detection [13]. It can be applied to various organs, such as lungs, liver, brain, bones, etc. Recent research declare that they provide efficient and accurate results in the area of segmentation. One of the studies performed the segmentation of lung tumors on the Lotus data set and obtained improved performance by considering a lightweight model design [14]. Convolutional neural networks can also be employed in different scenarios for the identification and classification of lung nodules [15,16].

Deep learning problems need a huge data set with accurate and reliable data. Sufficient amounts of reliable data images are necessary to obtain accurate results, but a lack of sufficiently large data sets, high costs to get data from sources, and the issues around dealing with personal information create difficulties in obtaining sufficient amounts of data. As a solution to this problem, many methods have been developed for availing large amounts of data. True transferring weight of a pre-trained model is one of the popular methods for achieving this goal to a certain extent [8,9]. Another method is data augmentation. Data augmentation is the process of increasing the amount of data available by adding copies of the available data with slight modifications [10]. By using generative adversarial networks, it is possible to generate data artificially. Generative adversarial networks can generate high quality images that can be used instead of real data. In the proposed model, a GAN [3] was used to generate CT images of lungs. Various studies related to this field use the image generation technique. CT images generated from the available authentic data resources have been used for various applications [17,18,19,20,21,22]. Detection and segmentation for lung nodules has been performed effectively by using the mask R-CNN algorithm [23]. Lung nodule segmentation for raw thoracic CT scans employed regional CNN effectively for developing a fully automated system for detection and segmentation [24]. Optimized 3D-UNET modeling has also been proposed to segment pulmonary nodules in CT images [25].

To achieve quality data images, a GAN was used. Authentic and accurate data samples were used here to generate the needed database. The finetuned architecture used real data samples to make the proposed system more stable and efficient. The methods employed, and other specifications, are included in the following sections in detail.

3. Materials and Methods

3.1. SegNet-Based Proposed Model

Semantic segmentation models, in a broad sense, comprise networks responsible for encoding and decoding. Here, the encoder works as a classification network, which is pre-trained. The encoder is composed of different convolution layers. The feature extraction process is done by the encoder, which is composed of convolution layers, as shown in Figure 1.

Features are taken out of the input through the convolution process. The decoder decodes low resolution feature maps to full input resolution for the purpose of dense classification. The decoder is composed of different deconvolution layers. The aim of the deconvolution layer is up-sampling. Through deconvolution, the features that are reduced in dimensions are up-sampled to image size. Figure 1 shows the SegNet system flow from the input to the output. In SegNet a set of encoder layers encodes the input images based on deep convolution. These images are decoded by corresponding decoder layers. Decoded images are forwarded to a classification layer. This layer acts as a pixel-based classifier. Feature fusion layers combine the features that are collected from the patches. Input images are sliced into patches in a horizontal and vertical fashion. These are passed via the encoder and feature fusion layers. Based on the feature characteristics, the decoder layers produce the output. The decoder output will be the lung nodule segmentation result, as shown in the Figure 1

3.2. Methodology Employed

As shown in the Figure 2, the data set consists of images that are fed into the GAN network. The network, which is adversarial, in nature processes the given data. Random input images are used as the input. The generator network generates realistic images from the available original data. Simultaneously, the discriminator network learns from the real and generated images. The learning will result in a conclusion having a unique result, termed as prediction. Throughout the learning process, different parameters are updated and finetuned. The performance of the SegNet model is evaluated and modified by different parameters for performance improvement. The model compiles and trains with real data images, and the performance is evaluated with a modified evaluation metric. Upon updating the loss function and optimizing the parameters, the performance is evaluated. The batch size and rate of learning are also adjusted to improve the performance. The performance is improved further by modifying the layers within the system [26]. The final segmented results are generated from the SegNet by transferring the weights.

The proposed system, shown in Figure 2, generates images using a generative adversarial network. After that, the generated images are used to learn a SegNet, and the parameters are updated by using the real data images. Through parameter transfer learning, the SegNet is updated for image segmentation. The results are verified by using a test data set.

The advantages of the proposed system are evident while looking into various factors and the performance of different architectures mentioned in the literature review. The systems using U Net face a lot of memory constraints, as features are changed into the expansion path. However, the SegNet offers an efficient architecture by using an encoder decoder network. When compared to the fully connected networks, the memory needs are less, and it can effortlessly be used without any memory constraints, even with large data sets. Similarly, when working with small datasets, the proposed SegNet method reduces the chances of overfitting in comparison with the other architectures. The overall design of SegNet can be considered a computationally efficient one that can be deployed in systems with resource constraints. The efficiency of the proposed SegNet architecture is significantly high as compared with other architectures having low efficiencies and so many of the other constraints mentioned above. Volumetric segmentation and analysis can be done by using V-Net. The analysis done here is in terms of volumetric parameters and is suitable for the three-dimensional deep net architecture.

3.3. Database and Setup

Many public databases are available with data sets of various images of internal organs for research purposes. Lung Nodule Analysis is a data set widely known as LUNA16, which has CT images of lungs. Another source is the Decathlon lung data set, having a lot of lung image data. NSCLC radio genomics are also helpful for this purpose. Data sets available with LUNA16 include three-dimensional CT images intended for lung nodule detection. The Decathlon includes three-dimensional CT images and detailed segmentation with labels. Images without pre-processing from the Decathlon lung data set were used to prepare three sets of training data sets. The NSCLC radio genomics data set includes non-small cell lung cancer CT and PET/CT images. Images with segmentation labels could only be used as test images in this system.

The LUNA16 lung database consists of 888 CT scans with slice thicknesses less than 2.5 mm. In the LIDC/IDRI database, the presence of nodules has three categories—no nodule, less than 3 mm nodule, greater than 3 mm nodule—executed through an annotation process. GAN can be used to generate images for lung nodule detection. The same method can be used for generating a dataset for nodule segmentation. Due to the lack of labels of large true nodules, generation of data is not practical. Three-dimensional CT images corresponding to small lung nodules were considered for the generation of images having cancerous nodules.

3.4. Augmented Image Generation Using GAN

The proposed system shown in Figure 2 generates images using a generative adversarial network. Instead of traditional square patches, we used horizontal and vertical patches. This kept global and local appearance features in the patches. A dimension of 16 × 64 and 64 × 16 was kept for horizontal and vertical patches, respectively. The mean intensity and variance of all the patches in the training set were extracted. All the patches were then normalized with zero mean and unit variance before being fed into the auto encoder for training.

The idea can be implemented by using a deep learning-based generative model. Generative adversarial networks (GANs) were considered here for improving system performance by training the system intelligently. The generative adversarial network consists of two opposing networks: one is generative, and the other is discriminative in nature. These two networks race to reach the learning target for obtaining optimized results. These networks can be used to generate any distribution of data through training. Random inputs were used here to generate the initial set of images. The discriminator discriminates the images and assists the generator in reducing the discrimination. Through the feedback, the generator modifies the parameters and tries to generate better outputs. This can be achieved by updating loss functions. Generator and discriminator loss are updated throughout the course of process, and thereby achieve resultant images that highly resemble real ones. Thus, the network provides data with high quality for the process. CT images can be generated for various applications using these generative models [17,18,19,20,21,22].

The expression for standard GAN loss function is shown here. It comprises loss function of discriminator and loss function of generator. Generator tries to improve the performance, and, thereby, the loss function gets minimized. However, the discriminator functions in the reverse direction by trying to maximize the function.

Loss function = Ex [log (D(x))] + EZ [log (1 − D (G(z)))]

(1)

New realistic data can be generated with great accuracy using generative models. Here, the probability distribution of the existing dataset is used to generate the new data set. Generative adversarial network comes under this category and is used here to generate realistic data. The performance can be improved by integrating multiple generators and discriminators within the system. The GAN can also be trained in a distributed fashion, and, then, it can be presented as a multi-discriminator GAN. In GAN, as shown in the Figure 3, random images are given as the input to the generator section. From the random inputs received, the generator tries to generate sample images. The purpose of the generator section is to generate images that are identical in nature with the real images. Thus, from the random inputs, the generator generates real images as samples. These samples are fed into the discriminator. The discriminator receives real image samples as well. The function of the discriminator is to discriminate generated images from the real images. The outputs of the discriminator section, known as loss functions, are generated and classified as generator loss function and discriminator loss function.

The signal provided by the discriminator functions acts as a feedback signal to the generator. This signal is used by the generator to update the weights. Through updating the weights of the generator, it generates images that are more identical to the real image samples. The aim of the generator is to generate images that are very much identical to the real images. This can be achieved through the updating of weights periodically. Back propagation from the discriminator helps the generator to update the weights and reproduce images with better similarity.

3.5. Transfer Learning Process

Accumulation of information does not happen in the case of traditional machine learning. Information that was learned previously is not considered in the case of the traditional learning process. Traditional learning is an isolated process and considered as a single task learning process. In transfer learning, the learning process relies on information that is learned previously. Hence, the transfer learning process can be done faster and require less amounts of data for training.

Transfer learning starts from taking a previously learned network for some other process. The previously learned information was used here in the new training process, which made it faster and more accurate. The method of reusing a pre-trained model was considered here. The information gained via training can be used for improving the performance of another model by transfer learning process. In practice, it is done by transferring the weights that are learned by the network. A problem can be solved easily by using the information learned from solving another problem, which is related to the first one. Figure 4 explains the concept of transfer learning.

The procedure followed throughout the investigation is described below.

3.6. Training and Optimization

Figure 5 shows some examples of benign and malignant nodules found in the lung tissues. The first row represents benign nodules while the second row shows the malignant nodules. All the images are taken from the LUNA16 lung database consisting of 888 CT scans. A GAN was used for generating further images from the dataset.

A total of 100, 1000 data images were generated for training the SegNet architecture for nodule segmentation. The real and generated images were considered for the training process. The details of training data for the GAN and SegNet are shown in

Table 1. Training the GAN—original and augmented images used for training.

Training	Benign	Augmented	Malignant	Augmented
SegNet	450	1280	430	1148
GAN	225	640	215	574

.

The original data from LUNA 16 was used to further generate benign and malignant samples using the GAN. A total of 440 images and their augmented images were used for training the GAN and generated 1,001,000 images. Test data included 300,000 images, which were a combination of the generated and real data images.

3.7. Model and Evaluation Process

The proposed method combined generative adversarial networks (GAN) and SegNet, two deep learning models that show promising results in accurately segmenting pulmonary nodules in CT images. The GAN was used to generate additional training data for the SegNet model, while SegNet was used for the actual segmentation task. This approach improves segmentation accuracy and reduces the need for large amounts of manually annotated data. The proposed GAN–SegNet model outperforms other deep learning models such as VAE, Autoencoder, and U Net in terms of accuracy and AUC. However, the training process can be time-consuming without a high-end GPU. The GAN is a deep learning architecture composed of a generator network and a discriminator network, trained together in a minimax game to produce high-quality synthetic data. SegNet is an encoder–decoder architecture specifically designed for pixel-wise classification, capable of accurately segmenting complex structures in medical images. The proposed model “GAN-Enhanced SegNet” or “G-Net” for short, reflects the fact that GAN enhances SegNet’s performance by generating additional high-quality training data. Overall, the GAN–SegNet model presents a promising approach for the automated detection and segmentation of pulmonary nodules in CT images, with potential applications in clinical settings.

There is also a significant difference in segmentation while using GAN-based images. This is evident from the evaluation table in

Table 2. Performance evaluation of SegNet + GAN.

Method	DSC	PPV	Sensitivity
SegNet + GAN + augmentation	0.87	0.86	0.89
SegNet + GAN	0.86	0.85	0.87
SegNet + augmentation	0.80	0.79	0.82
SegNet	0.78	0.77	0.80

. We mainly used three parameters for evaluation, namely, dice similarity coefficient (DSC), positive predictive value (PPV), and sensitivity. DSC is a measure of overlap between ground truth and segmented results. Sensitivity shows how much positive proportions are measured correctly. Positive results are represented as a proportion as PPV. The parameters are evaluated from confusion matrix as follows:

DSC = 2TP/(FP + 2TP + FN)

(2)

PPV = TP/(TP + FP)

(3)

Sensitivity = TP/(TP + FN)

(4)

Then, the receiver operating characteristic (ROC) curve of SegNet, during training the model with GAN and without GAN, was also evaluated. The area under the curve was calculated with the ROC curve.

4. Results and Discussion

Figure 6 shows the sample images generated by the generative adversarial networks. Benign nodules are more round in shape compared to malignant images generated with spicula and withdrawal into the pleura. The images generated by GAN cannot be differentiated from the original images, even by a trained radiologist. This proves the importance of images generated by GAN.

Nevertheless, the generated images are low resolution 64 × 64 in size. GAN is able to generate 1024 × 1024 resolutions for natural images. The improvement in GAN-based architectures will improve the resolution of the medical images, which will further enhance the accuracy in the detection and segmentation of lung nodules. Top row of the Figure 7 represents the SegNet segmentation on pulmonary nodules for benign cases along with its ground truth counterpart. Since the benign nodules are more round in shape, more accurate segmentation is possible compared to malignant cases, as is shown in Figure 7, bottom row In malignant cases, the nodules extend to the lung bones, and it is difficult to differentiate the difference. This reduces the segmentation accuracy considerably whencompared to benign nodules.

Different methods and the performance results are tabulated in Table 2. It is evident from the table that GAN along with image augmentation was clearly the winner. Simultaneously, GAN with SegNet showed very small deviation from the winner. This clearly shows the advantage of using GAN for generating images for nodule segmentation. The method was also compared with other encoder–decoder architectures.

The performance comparison is given in

Table 3. Performance comparison of Segnet with GAN with other encoder–decoder architectures.

Method	DSC	PPV	Sensitivity
SgNet + GAN	0.87	0.86	0.89
U-Net + GAN	0.84	0.86	0.87
Autoencoder	0.75	0.74	0.77
Variational Autoencoder (VAE)	0.80	0.81	0.82

. Finally, the receiver operating characteristic (ROC) curve of SegNet while training the model with GAN and without GAN is given in Figure 8. The area under the curve (AUC) for SegNet + GAN was found to be 0.851, while that of SegNet was 0.76.

Based on the proposed method and the methodologies employed, the effect of several design choices on the performance was analyzed. Here, the important factors that were analyzed, along with their corresponding impact on the performance of the model, are tabulated in

Table 4. Design choices and the impact on the performance of the system under investigation.

Design Choice	Impact on Performance
SegNet architecture	Deeper model with more convolutional layers performed better. For example, SegNet-4 with 11 convolutional layers outperformed SegNet-2 with 7 convolutional layers.
Adversarial training	Significant improvement in nodule segmentation accuracy. The proposed adversarial training method improved the mean dice coefficient for nodules from 0.717 to 0.773.
Pre-processing	Lung cropping and normalization improved nodule segmentation accuracy. The mean dice coefficient for nodules increased from 0.717 to 0.743 after lung cropping and normalization.
Loss function	Dice loss performed better than binary cross-entropy. The mean dice coefficient for nodules was higher with dice loss (0.773) than with binary cross-entropy (0.752).

.

5. Conclusions

The proposed work, with the help of the available data set, managed to generate sufficient amounts of data with the help of GAN. Perfect samples were considered here for the generation of data. Hence, the data generated assured the quality that was used for training the network. For training the GAN, 440 images and augmented images were used. A total of 1,001,000 images were generated and used for training the SegNet architecture. The segmentation showed improved accuracy in the case of benign samples, as it had a perfect shape compared to malignant samples. Through the evaluation of the parameters, it is clear that the GAN with image augmentation showed better performance.

The evaluation is clearly presented in the tabulation with comparison of parameters. The proposed model outperformed the SegNet model by only 9% and the GAN with SegNet model by 2%. Different encoder–decoder combinations were also considered here, and the comparison of parameters is tabulated in Table 3. SegNet with GAN performed well in comparison with VAE, Autoencoder, and U Net.

The proposed model outperformed the VAE scheme by 7% and the U Net with GAN by 2%. The receiver operating curve was also plotted here to validate the result. SegNet and SegNet with GAN were plotted for comparing the AUC. The AUC of the proposed scheme outperformed the former.

The time needed for the training process was very high with the i5 processor with 3.2 Ghz clock speed and 8 GB RAM, which was used in the initial stages of the study. The processing time was reduced when the processor configuration switched to a higher level. The processing time can be reduced to 120 min if it is assisted by a GPU (NVIDIA RTX 4090) integrated within the system.