UMGAN: Underwater Image Enhancement Network for Unpaired Image-to-Image Translation

Abstract

Due to light absorption and scattering underwater images suffer from low contrast, color distortion, blurred details, and uneven illumination, which affect underwater vision tasks and research. Therefore, underwater image enhancement is of great significance in vision applications. In contrast to existing methods for specific underwater environments or reliance on paired datasets, this study proposes an underwater multiscene generative adversarial network (UMGAN) to enhance underwater images. The network implements unpaired image-to-image translation between the underwater turbid domain and the underwater clear domain. It has a great enhancement impact on several underwater image types. Feedback mechanisms and a noise reduction network are designed to optimize the generator and address the issue of noise and artifacts in GAN-produced images. Furthermore, a global–local discriminator is employed to improve the overall image while adaptively modifying the local region image effect. It resolves the issue of over- and underenhancement in local regions. The reliance on paired training data is eliminated through a cycle consistency network structure. UMGAN performs satisfactorily on various types of data when compared quantitatively and qualitatively to other state-of-the-art algorithms. It has strong robustness and can be applied to various enhancement tasks in different scenes.

1. Introduction

Underwater imaging is a significant source of oceanic information, advancing the study of marine biology, marine archaeology, marine ecology, and naval military [1,2]. The effects of absorption and scattering on the underwater imaging process lead to significant image deterioration with low contrast, color distortion, blurred details, and uneven illumination [3]. Degraded images seriously affect underwater vision tasks and research. It is of great scientific significance to improve the visual quality of original underwater images with the help of image processing technology. However, the degree of deterioration and the existence of issues differ from one underwater image to the next. Therefore, underwater image enhancement for different scenes is a challenging task.

To address the issue of underwater images, researchers are currently employing a variety of techniques, including contrast enhancement [4,5], frequency domain enhancement [6,7], color constancy [8,9], and image fusion [10,11]. However, these techniques frequently focus on a single data type. They have low robustness and do not handle different styles of underwater images very well. With the development of image processing technology [12,13,14,15,16], deep learning techniques have advanced significantly in the field of image processing in recent years. An increasing number of researchers are applying deep learning methods to underwater image enhancement [17,18,19,20,21]. Compared with traditional methods, deep learning methods trained with enormous amounts of data can better handle images in different scenes.

A generative adversarial network called UMGAN has been proposed for the enhancement of several underwater scene images. GAN is adopted to build unpaired mappings between turbid and clear underwater image spaces, which frees us from the reliance on precisely paired images. According to the characteristics of the underwater environment, a feedback mechanism for underwater image quality measurement is proposed. Based on the quality of the created images, feedback is provided to the generator to continuously optimize the generated results. A noise reduction network is designed to prevent the creation of image noise during the training process since artifacts and texture noise are frequently present in the images produced by GAN. In addition, dual discriminators are used to balance global and local enhancements. The issues of over- and underenhancement in local regions are avoided by adaptive modification of the enhancement effect in each region. Figure 1 shows the underwater images of different scenes and enhancement effects.

The main contributions of this work are summarized according to three aspects. First, a feedback mechanism for the characteristics of underwater images is proposed. The generator is optimized by providing constant feedback about the image quality, achieving image enhancement for a variety of underwater scenes. Second, the noise reduction network is designed. The problem of artifacts and textural noise in the images generated by GAN is solved. Third, a global–local discriminator is employed. Dual discriminators weight the image results and adaptively enhance local regions while enhancing the global image.

2. Related work

2.1. Underwater Image Enhancement

When light is transmitted in water, it is affected by both absorption and scattering. Therefore, underwater images often suffer from low contrast, color distortion, blurred details, and uneven illumination [22]. Two contrast enhancement methods, ICM [23] and UCM [4], were proposed by Iqbal et al., who were the first to research underwater image enhancement. Inspired by Iqbal et al., a two-stage technique for underwater image contrast enhancement and color correction was presented by Ghani et al. [5]. A series of histogram equalization methods (HE [24], AHE [25], and CLAHE [26]) was also used to enhance underwater images [27,28]. Jin et al. [29] combined CLAHE with Gaussian differential pyramids to solve the problem of low-contrast and blurred details in underwater images.

Frequency domain techniques such as Fourier transform and wavelet transform are used by researchers for underwater image enhancement [6,7]. Retinex is a widely used technique for enhancing images that is based on scientific research and analysis. Extended multiscale retinex was utilized by Zhang et al. [8] to enhance underwater image quality. Zhou et al. [30] used multiscale retinex to extract the light components to adjust the three channels for color correction. Image fusion entails performing several picture improvement procedures on the source image before merging the various improved images by weighting. Following the adjustment of the white balance, Ancuti et al. [10] performed a multiscale fusion of underwater images. A combination of enhanced background filtering and wavelet fusion methods was used by Ghani et al. [11]. Then, they integrated homomorphic filtering, recursive overlapping CLAHS, and dual-image wavelet fusion for underwater image enhancement [31]. Lei et al. [32] implemented underwater image enhancement based on color correction and dual-image multiscale fusion.

2.2. Deep Learning Methods

In recent years, an increasing number of researchers has applied deep learning methods to underwater image enhancement. Depending on the network model used, deep-learning-based underwater image enhancement methods can be classified as convolutional neural network (CNN)-based or generative adversarial network (GAN)-based methods. Wang et al. [33] first applied CNN to underwater image enhancement. Li et al. proposed Water-Net [21] and UWCNN [20] to implement end-to-end underwater image enhancement. To better support the deployment of underwater exploration missions on portable devices, Shallow-UWnet [34] maintains performance while reducing the number of participants.

WaterGAN [17] is an example of the early use of GAN for underwater image enhancement. It eliminates the reliance on training data, and the model synthesizes in-air RGB-D images and depth maps into underwater images. Researchers have focused on the application of GAN in underwater image enhancement from two directions: a conditional generative adversarial network (cGAN) and a cycle-consistent generative adversarial network (CycleGAN). Guo et al. [35] proposed a multiscale dense GAN for underwater image enhancement. Zhou et al. combined the physical model with GAN and used an underwater imaging model to simulate an underwater training dataset from RGB-D data. Inspired by CycleGAN [36], a weakly supervised color-transfer method was proposed by Li et al. [18] to solve the color distortion problem in underwater images. On this basis, Lu et al. [19] proposed a multiscale CycleGAN (MCycleGAN) by combining the dark channel prior and CycleGAN. FUnIE-GAN [37] implements real-time underwater image enhancement. Du et al. [38] added a content-loss regularizer and a blur-promoting adversarial loss regularizer to CycleGAN to retain detailed information about the enhanced images.

3. Materials and Methods

An underwater multiscene generative adversarial network (UMGAN) is proposed for the purpose of underwater image enhancement. In the forward process, the turbid underwater image is converted to a clear image by the G generator and the noise reduction network. A quality feedback mechanism was designed. The generated images feed the image quality back to the generator to continuously optimize the generated results. Then, the global–local discriminator is developed to jointly determine the effect of the generated image. In the same way, the F generator maps the clear image domain to the turbid image domain. SSIM loss is also used to control the consistency of the image structure in both domains. The overall framework is shown in Figure 2.

3.1. Underwater Image Quality Measure Feedback

A non-reference underwater image quality measure (UIQM) was proposed by Panetta et al. [39]. The UIQM includes three image attribute measures: the underwater image colorfulness measure (UICM), the underwater image sharpness measure (UISM), and the underwater image contrast measure (UIConM). Inspired by Panetta et al.’s work, a feedback mechanism for underwater images is proposed. The UIQM loss function is designed to evaluate the quality of the generated images in terms of colorfulness, sharpness, and contrast, which is fed back to the generator to optimize the generated image results. During the training process, G mapping continuously receives feedback from the UIQM loss function, and the quality of the generated images is gradually improved. In the forward process, the UICM, UISM, and UIConM values of the generated images are fed back to the generation module. The UIQM loss of the image is expressed as follows:

$$L_{UIQM}\left(I,G\right)=\frac{1}{n}{\displaystyle \sum }_{pix=1}^n\left({\lambda }_1-L_{UICM}\left(G\left(I_{pix}\right)\right)\right)+\frac{1}{n}{\displaystyle \sum }_{pix=1}^n\left({\lambda }_2-L_{UISM}\left(G\left(I_{pix}\right)\right)\right)+\frac{1}{n}{\displaystyle \sum }_{pix=1}^n\left({\lambda }_3-L_{UIConM}\left(G\left(I_{pix}\right)\right)\right)$$

(1)

where $I$ denotes the input turbid image, $G\left(·\right)$ denotes the enhanced output of the generator, $pix$ denotes each pixel of the image, and $n$ denotes the total number of pixels. ${\lambda }_1,{\lambda }_2$, and ${\lambda }_3$ are set to 1, 2.5, and 2, respectively, as determined by extensive experiments on the training dataset.

3.2. Noise Reduction Network

The images generated by GAN networks frequently include noise. In this case, a noise reduction network is proposed, which is added after the generator. Before being predicted by the discriminator, the image matrix created during each training round is sent into the noise reduction network, which improves the quality of the generated images in each round. The network’s filter is based on Gaussian filtering. A 3 × 3 Gaussian kernel is used to convolve the generated images. The noise reduction network can effectively improve the problem of severe textural noise in the images generated by the GAN. A visual comparison after adding the noise reduction network is shown in Figure 3.

3.3. Global–Local Discriminators

Different scenes in a picture are located at different distances from the camera. Light conditions also differ depending on the region. Global discriminators alone often cannot adaptively enhance every local region in an image. Inspired by previous work [40], a global–local discriminator structure for underwater image enhancement is designed. It adaptively enhances local areas while enhancing the global image. The original global discriminator judges the full map enhancement effect. The local discriminator crops out random local blocks from the generated image for determination. This method avoids the problems of over- and underenhancement of local areas. The comprehensive results of the two discriminators are:

$$D\left(I\right)=\alpha D_{Global}\left(I\right)+{\displaystyle \sum }_{k=1}^n{\beta }_k{D}_{Local}\left(I_k^{ij}\right)$$

(2)

where $\alpha$ and ${\beta }_k$ are the weights of the two discriminators, and $\alpha +{\displaystyle \sum }_{k=1}^n{\beta }_k=1$. $i$ and $j$ denote the height and width of the randomly cropped block, respectively ($i=64$, $j=64$). To ensure the stability of the cycle structure, local discriminators are introduced in both the forward and backward processes. The visual comparison after adding the global–local discriminator is shown in Figure 4.

3.4. Loss Function

UMGAN can realize underwater image enhancement by learning unpaired datasets, which eliminates the dependence on paired training data. For this purpose, a set of loss functions are proposed to evaluate the effect of image enhancement. The following four loss functions are used to train the network.

3.4.1. UIQM Loss

UIQM is introduced to measure the three aspects of colorfulness, sharpness, and contrast of the generated images. The evaluation results are fed back to the generator to optimize the generation results. Since only the enhanced image generated from the blurred image needs to be enhanced, the UIQM loss is only applied in the forward process. Referring to Equation (2), the UIQM loss of the algorithm is expressed as follows:

$$L_{UIQM}\left(X,G_{X2Y}\right)=\frac{1}{n}{\displaystyle \sum }_{pix=1}^n\left({\lambda }_1-L_{UICM}\left(G\left(X_{pix}\right)\right)\right)+\frac{1}{n}{\displaystyle \sum }_{pix=1}^n\left({\lambda }_2-L_{UISM}\left(G\left(X_{pix}\right)\right)\right)+\frac{1}{n}{\displaystyle \sum }_{pix=1}^n\left({\lambda }_3-L_{UIConM}\left(G\left(X_{pix}\right)\right)\right)$$

(3)

3.4.2. Adversarial Loss (Global–Local)

Adversarial loss is used to train the generators and discriminators to make the generated images more realistic. The loss function at the global level is as follows:

$$L_{GAN}^{Global}\left(G,D_Y,X,Y\right)={\mathbb{E}}_{y~p_{data}\left(y\right)}\left[log{D}_Y\left(y\right)\right]+{\mathbb{E}}_{x~p_{data}\left(x\right)}\left[log(1-D_Y\left(G\left(x\right)\right)\right]$$

(4)

The mapping function ($F:Y\to X$) and its discriminator ($D_X$) are introduced with a similar adversarial loss, i.e., $L_{GAN}^{Global}\left(F,D_X,Y,X\right)$. The loss function on the local discriminator is:

$$L_{GAN}^{Local}\left(G,D_L,X,Y\right)={\mathbb{E}}_{y~p_{data}\left(y\right)}^{y^{ij}\in y}\left[log{D}_L\left(y^{ij}\right)\right]+{\mathbb{E}}_{y~p_{data}\left(y\right)}^{y^{ij}\in G\left(x\right)}\left[log(1-D_L\left(G\left(x\right)\right)\right]$$

(5)

Referring to Equation (3), the total adversarial loss is:

$$L_{GAN}=\alpha \left[L_{GAN}^{Global}\left(G,D_Y,X,Y\right)+L_{GAN}^{Global}\left(F,D_X,Y,X\right)\right]+{\displaystyle \sum }_{k=1}^n{\beta }_k\left[L_{GAN}^{Local}\left(G,D_L,X,Y\right)+L_{GAN}^{Local}\left(F,D_X,Y,X\right)\right]$$

(6)

3.4.3. Cycle Consistency Loss

The space of potential mapping functions is constrained by cycle consistency loss. In the forward process, each image ($x$) from domain $X$ can be transformed to itself after mapping $G$ and mapping $F$, i.e., $x\to G\left(x\right)\to F\left(G\left(x\right)\right)\approx x$. Similarly, $G$ and $F$ should satisfy the backward cycle consistency, i.e., $y\to F\left(y\right)\to G\left(F\left(y\right)\right)\approx y$. The cycle consistency loss is:

$$L_{cyc}\left(G,F\right)={\mathbb{E}}_{x~p_{data}\left(x\right)}\left[F\left(G\left(x\right)\right)-x_1\right]+{\mathbb{E}}_{y~p_{data}\left(y\right)}\left[G\left(F\left(y\right)\right)-y_1\right]$$

(7)

3.4.4. SSIM Loss

There is a strong correspondence between the enhanced image and the original image. SSIM loss preserves the content and structure between them [18]. The formula for SSIM is:

$$SSIM\left(p\right)=\frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x{}^2+{\mu }_y{}^2+C_1}·\frac{2{\sigma }_{xy}+C_2}{{\sigma }_x{}^2+{\sigma }_y{}^2+C_2}$$

(8)

where p is the center pixel, and $x$ and $y$ are the image patches in $X$ and $G\left(x\right)$, respectively. A 13 × 13 sliding window is used for convolutional filtering. ${\mu }_x$ is the mean of $x$, ${\sigma }_x$ is the standard deviation of $x$, and ${\sigma }_{xy}$ is the covariance of $x$ and $y$.$C_1=0.02$, and $C_2=0.03$. SSIM losses can be expressed as follows:

$$L_{SSIM}\left(x,G\left(x\right)\right)=1-\frac{1}{n}{\displaystyle \sum }_{pix=1}^n\left(SSIM\left(p\right)\right)$$

(9)

where n is the number of pixels. SSIM losses are calculated in both the forward and backward processes. Similarly, the SSIM loss in the backward process is $L_{SSIM}\left(y,F\left(y\right)\right)$.

The total loss function is a linear combination of the above four losses and is expressed as follows:

$$Loss=L_{UIQM}+L_{GAN}+L_{cyc}+L_{SSIM}$$

(10)

4. Discussion

4.1. Dataset and Implementation Details

Since UMGAN has the unique ability to train using unpaired underwater turbid/clear images, it is possible to collect unpaired training sets covering different image qualities and contents. A large number of training samples ensures the performance of the model and avoids overfitting. The UIEB [21] and EUVP [37] datasets are numbered by natural numbers starting from 1. The entire dataset totals 15,980 clear images and 18,405 blurred images. The images for training were selected using the method of generating random numbers. We used the entire dataset for training, and the training time exceeded 72 h. After comparing the experimental results, it was found that there was no significant difference between the training effect of the randomly selected images and the full dataset, meaning that the amount of data from the randomly selected images is enough to guarantee the effectiveness of the algorithm. To ensure the efficiency of completing extensive training during the subsequent ablation experiments and comparison experiments, we adopted a random selection of images. The final selection from the dataset was 2397 underwater turbid images and 2635 clear images. All input images were converted to PNG format and resized to 256 × 256 × 3 pixels. Training dataset samples are shown in Figure 5.

First, 100 epochs are trained from scratch with a learning rate of ${10}^{-4}$. Then, another 100 epochs are performed, and the learning rate decays linearly to 0. The learning rate is set to ${10}^{-4}$ because there are a number of parameters in the algorithm that affect the final result of the image. These parameters interact with each other. The lower learning rate ensures that we do not miss any local minimum values. This avoids overadjustment of one parameter in one training session, which could lead to large changes in other parameters. Using the Adam optimizer, the batch size was set to 16. The entire training process took 14 h on four Nvidia 2080Ti GPUs.

4.2. Ablation Study

To demonstrate the effectiveness of each component in the algorithm, several ablation experiments were performed. Specifically, three experiments were designed by removing the components of UIQM loss, noise reduction network, and local discriminator, respectively. The effects of the noise reduction network and the local discriminator are demonstrated in Figure 3 and Figure 4, respectively. In Figure 6, the first row shows the input images. The second row shows the effect of the algorithm in enhancing the image without UIQM loss. The third row shows the effect of the algorithm in enhancing the image without the noise reduction network. The fourth row shows the enhanced image using only the global discriminator. The last line is produced by the final version of UMGAN. The color enhancement of the images is significantly reduced after removing UIQM losses. After the noise reduction network is removed, more noise and artifacts are produced. After reducing the local discriminator, the enhancement effect of some edge details is insufficient.

Full-reference image quality evaluation metrics such as MSE, PSNR, and SSIM and no-reference image quality evaluation metrics such as UCIQE and UIQM were used to objectively evaluate the underwater image enhancement effect. A total of 300 underwater images of different styles were used to evaluate the results of the ablation experiments. As shown in

Table 1. Quantitative evaluation of ablation experiments (best scores are bolded; the numbers in parentheses represent the percentage boost after adding the module).

Metric	Without UIQM Loss	Without Noise Reduction Network	Without Local Discriminator	UMGAN
MSE	532.58 (1.24%)	678.45 (28.97%)	561.63 (6.76%)	526.05
PSNR	23.33 (0.94%)	21.48 (9.64%)	23.07 (2.08%)	23.55
SSIM	0.8934 (0.68%)	0.8214 (9.51%)	0.8915 (0.90%)	0.8995
UCIQE	0.4208 (4.66%)	0.4372 (0.73%)	0.4232 (4.06%)	0.4404
UIQM	3.191 (9.40%)	3.268 (6.82%)	3.193 (9.33%)	3.491

, UMGAN showed the best MSE, PSNR, SSIM, UCIQE, and UIQM scores.

4.3. Qualitative Evaluation

A comparison was made between our algorithm and several other algorithms. The results are shown in Figure 7. The first row shows the turbid underwater images. The second to tenth rows show the enhanced images obtained by the UCM algorithm [4], the CLAHE algorithm [26], the white balance algorithm [41], the retinex algorithm [9], the image fusion algorithm [10], the UWCNN algorithm [20], the CycleGAN algorithm [36], the FUnIE-GAN algorithm [37], and the Shadow-UWnet algorithm [34], respectively. The last line shows the results generated by our proposed UMGAN.

The contrast of the enhanced images was improved to some extent using the UCM method. Colors such as red and green are more prominent. However, the image is still blurred, and the defogging effect is poor. The CLAHE method considerably optimizes the color cast as compared to the UCM method. The distribution of red, green, and blue pixels is equal, but little noise and few artifacts are introduced. The white balance method highlights the red color, but the images’ overall color is dark, and some of them have more artifacts. The retinex method improves image brightness and contrast while introducing a tiny degree of artifacts. Over- and underenhancement issues are successfully resolved by the image fusion technique. The image contrast is high, and the color cast is greatly improved, but the colors are not very vivid. UWCNN trained the weight files for a variety of scenarios, and the red color was significantly enhanced, but the results were all darkened. The CycleGAN method is better at restoration, but overenhancement produces artifacts and textural noise. There are artifacts in the images generated by FUnIE-GAN. Shallow-UWnet has fewer parameters and runs fast, but the output image is low in pixels. In contrast, UMGAN has the best overall visual appearance. In addition, in the detail processing step, UMGAN not only adaptively enhances each region of the image but also reduces artifacts and noise.

4.4. Quantitative Evaluation

A total of 300 underwater images of different styles were selected for quantitative analysis of the above method. As shown in

Table 2. Quantitative evaluation results of several algorithms (best scores are bolded).

Method	MSE	PSNR	SSIM	UCIQE	UIQM
UCM	725.47	21.82	0.8863	0.4114	3.0164
CLAHE	1924.31	15.67	0.5573	0.4528	2.9699
White balance	8238.24	10.61	0.4266	0.3089	2.3813
Retinex	2873.83	14.18	0.6353	0.4417	3.1860
Image fusion	1884.54	16.75	0.7336	0.4202	3.3534
UWCNN	1959.92	16.22	0.8095	0.3498	3.1654
CycleGAN	1150.65	19.05	0.7431	0.4587	3.3417
FUnIE-GAN	952.22	19.15	0.7481	0.4224	3.4052
Shadow-UWnet	740.95	20.15	0.8460	0.3758	3.0890
Ours	526.05	23.55	0.8995	0.4404	3.4911

, among all the methods, UMGAN showed the best MSE, PSNR, SSIM, and UIQM scores, improving on the second-place finisher by 27.49%, 7.93%, 1.49%, and 2.57%, respectively. The proposed achieved good overall performance.

4.5. Results and Significance

The UMGAN algorithm is trained and tested using publicly available datasets. Ablation experiments demonstrate the effectiveness of each module of the algorithm. UIQM loss improves the performance of the algorithm to enhance images. The noise reduction network removes artifacts and noise from the image. The global–local discriminator adaptively adjusts local region image effects. In the qualitative evaluation, UMGAN demonstrates the best visualization. Artifacts and noise are eliminated. In the quantitative analysis, UMGAN has the best overall score. Among them, MSE, PSNR, SSIM, and UIQM scored first. According to testing of different types of images in several datasets, UMGAN can enhance different types of underwater images very well. It solves the common problems of greenish, yellowish, and blueish images; low light; white fog; and low contrast of underwater images. The problem of artifacts and textural noise in the images generated by GAN is also solved by the designed module. UMGAN can be successfully applied to various types of underwater image enhancement tasks.

5. Conclusions

An image enhancement network called UMGAN, which is an end-to-end network trained by unpaired datasets, is proposed for a variety of underwater scenes. To force the generator to produce clear, high-quality, noise-free underwater images, an image quality feedback mechanism and a noise reduction network were designed. A global–local discriminator was created to ensure the enhancing effect in the local region using adaptive weighting loss to balance the impact of the two discriminators. Our algorithm’s performance was demonstrated through an ablation study. The results of qualitative and quantitative analysis show that our algorithm outperforms other algorithms in the processing of various styles of underwater images. In the future, we will attempt to improve the algorithm’s performance in terms of image detail information retention and training speed.