Series-Parallel Generative Adversarial Network Architecture for Translating from Fundus Structure Image to Fluorescence Angiography

Abstract

Although fundus fluorescein angiography (FFA) is a very effective retinal imaging tool for ophthalmic diagnosis, the requirement of intravenous injection of harmful fluorescein dye limits its application. As a screening diagnostic method that reduces the frequency of intravenous injection, a series-parallel generative adversarial network (GAN) architecture for translating fundus structure image to FFA images is proposed herein, using deep learning-based software that only needs an intravenous injection for the training process. Firstly, the fundus structure image and the corresponding FFA images of three phases are collected. Secondly, our series-parallel GAN is trained to translate FFA images from fundus structure image with the supervision of FFA images. Thirdly, the trained series-parallel GAN model is used to translate FFA images by only using fundus structure image. By comparing the FFA images translated by our algorithm, Sequence GAN, pix2pix, and cycleGAN, we show the advancement of our algorithm. To further confirm the advancements of our algorithm, we evaluate the peak signal-to-noise ratio (PSNR), structural similarity (SSIM) index, and mean-squared error (MSE) of our algorithm, Sequence GAN, pix2pix, and cycleGAN. To demonstrate the performance of our method, we show some typical FFA images translated by our algorithm.

1. Introduction

Retinal circulation imaging is critical not only for ophthalmic diagnosis but also for studying eye diseases [1,2]. The most popular ocular angiography methods include fundus fluorescein angiography (FFA) with narrow or normal field [3], wide field [4] and ultra-wide field [5,6,7], indocyanine green angiography (ICGA) [8], and optical coherence tomography angiography (OCT-A) [9]. The ICGA is primarily used for choroidal blood flow imaging in these methods, and OCT-A cannot show the blood filling process or static leakage, so FFA is the most accurate retinal circulation imaging tool. However, intravenous injection of dye is a necessity for FFA, which may cause adverse effects, including nausea, vomiting, syncope, shock, and even death in the worst-case scenario. Because of their weakened bodies, the elderly, who constitute the majority of the FFA population, are more likely to suffer adverse effects.

One way to alleviate the aforementioned problem is to reduce the frequency of FFA usage. Thus, several studies have been aimed at the development of FFA images translated from fundus structure image since 2018 with paired data [10,11,12,13,14], unpaired data [15,16,17], paired or unpaired data [18], and one-to-three data [19]. Most of these studies [10,11,12,13,14,15,16,17,18] focus on translating fundus structure image to late phase FFA images. Although the late phase FFA image is critical for ophthalmic diagnosis and disease research, whether or not the filling time of retinal vessels is normal is an important diagnostic basis for retinal vascular occlusion diseases [19], such as central retinal artery occlusion [20] and retinal vein occlusion [21].

Translating fundus structure image to more phases of FFA images is essential to provide information about the filling process of retinal vessels. One approach is to use several independent generative adversarial networks (GANs) to parallelly translate fundus structure image to their corresponding phases of FFA images. Although this method is direct and effective for this task, it does not make good use of mutual information between the different phases of FFA images, resulting in the algorithm’s low efficiency. To use the mutual information between the different phases of FFA images, Li et al. [19] proposed Sequence GAN that connects three GANs in series to translate fundus structure image to three phases of FFA image. However, the first translated FFA image still suffered from a low utilization rate of mutual information.

In this paper, we present a series-parallel GAN architecture for translating fundus structure image to fluorescence angiography to improve the utilization rate of mutual information between the different phases of FFA images. The proposed algorithm, called SPFSAT (Series-Parallel Fundus Structure to Angiography Translation), uses a series-parallel architecture-based generator and a single discriminator to achieve a high utilization rate of mutual information between the different phases of FFA images. In the series-parallel architecture-based generator, three decoders share the same encoder that extracts all the features including mutual information from the input fundus structure image at one time with the supervision of all the three phases labeled FFA images. Instead of using three parallel discriminators to judge three FFA images independently [19], the single discriminator judges the three FFA images together, which improves the utilization rate of mutual information between the three phases of FFA images. To show the advancements of our algorithm, we make a comparison among FFA images translated by our algorithm, Sequence GAN [19], pix2pix [22], and cycleGAN [23]. In detail, the typically translated FFA images are shown and the peak signal-to-noise ratio (PSNR), structural similarity (SSIM) index [24], and mean-squared error (MSE) are evaluated. Some typical FFA images translated by our algorithm are shown to demonstrate the performance of our method.

2. Methods

This section explains our SPFSAT algorithm, which is depicted in Figure 1. An encoder is used to extract the features of the input fundus structure image, and three decoders with different weights are used to generate three different fake FFA images from the encoder’s extracted features. To discriminate the real FFA images from the fake FFA images, the fundus structure image and three-phase FFA images are inputted to a single discriminator. Thus, the single discriminator judge FFA images not only by single FFA image information but also by mutual information among fundus structure image and three-phase FFA images. The loss function is determined by adding L1 loss and adversarial loss between real FFA images and fake FFA images to optimize the network performance.

2.1. Dataset Preparation

The datasets used in this study were created using the data of 66 patients (62 patients with eye diseases and 4 normal patients) with an image size of 768 × 768 pixels using Spectralis HRA equipment (Heidelberg Engineering, Heidelberg, Germany) between August 2011 and June 2019 at the Third People’s Hospital of Changzhou (Jiangsu, China). The fundus structure images were acquired at the 785 nm wavelength with a probing power of 1.2 mW, and the FFA images were acquired at the 488 nm wavelength with a probing power of 1.4 mW. The field of views of these images included 30°, 45°, and 60°. The image capture operating procedure involved system parameters selection, head placement, alignment of the imaging system, eye fixation, focusing, and pressing the shutter. The three-phase FFA images were captured from the arterial phase (11 to 15 s), venous phase (16 to 20 s), and late phase (5 to 6 min). Thus, we created one image group, which contains fundus structure image and the corresponding three-phase FFA images, for each patient. In each group, three-phase FFA images were manually registered to fundus structure image through translation and rotation, respectively. Figure 2 shows an example of a manually registered image group that contains the fundus structure image and the corresponding three-phase FFA images.

We used 51 image groups for the training set, 2 image groups for the validation set, and the remaining 13 image groups for the test set. Because the deep learning network requires a large dataset, we augmented the registered dataset with the data. The training set was generated by randomly cropping images with the image size of 512 × 512 pixels from 51 image groups during the training process. The validation set was generated by randomly cropping 50 images with the image size of 512 × 512 pixels from 2 image groups. The test set was generated by randomly cropping 100 images with the image size of 512 × 512 pixels from 13 image groups, and final testing results with different fields of view are generated by randomly cropping from these images. Because the diseased areas appear in different locations in the FFA images and are not very large, the augmented dataset that is generated by random cropping is balanced—nearly half of the images contain diseased areas and nearly half do not.

2.2. Deep Convolutional Neural Networks

The network architecture of our encoder is showed in Figure 3. This encoder is similar to the pix2pix encoder [22]. Specifically, the input image is sequentially operated by one convolution block, two downsample blocks, and six Resnet blocks to extract the features of the input image.

The network architecture of our decoder, which is in the same style as the decoder of pix2pix [22], is shown in Figure 4. To generate fake FFA images, it applies two upsample blocks, one convolution, and one hyperbolic tangent sequentially to the features extracted by the encoder.

Figure 5 shows the network architecture of our discriminator. It is distinguished by taking as input the $Fake$ enated fundus structure image and the corresponding three-phase FFA images. Thus, the discriminator judge FFA images not only by single FFA image information but also by mutual information among fundus structure image and three-phase FFA images. Then, this input is operated by four convolution blocks and one convolution sequentially.

2.3. Loss Function

For training our networks to output realistic fake FFA images, our loss function is built by the addition of L1 loss and adversarial loss between real FFA images and fake FFA images:

$$\mathrm{Loss}=L1 \mathrm{Loss}+\mathrm{Adversarial} \mathrm{Loss}$$

(1)

L1 loss is given by

$$L1 \mathrm{Loss}={{\displaystyle \sum }}_{Phase=1}^3{{\displaystyle \sum }}_{i,j}║{\mathrm{Fake} \mathrm{FFA}}_{Phase}(i,j)-{\mathrm{Real} \mathrm{FFA}}_{Phase}{(i,j)║}_1.$$

(2)

Adversarial loss is given by:

$$\mathrm{Adversarial} \mathrm{Loss}=║G_0-\mathrm{zeros}\_\mathrm{like}{(G_0)║}_2+║G_1-\mathrm{ones}{(G_0)║}_2,$$

(3)

in which

$$G_0=\mathrm{Discriminator}(\mathrm{concat}(\mathrm{Input},{ \mathrm{Fake} \mathrm{FFA}}_1,{ \mathrm{Fake} \mathrm{FFA}}_2,{\mathrm{Fake} \mathrm{FFA}}_3)),$$

(4)

$$G_1=\mathrm{Discriminator}(\mathrm{concat}(\mathrm{Input},{\mathrm{Real} \mathrm{FFA}}_1,{\mathrm{Real} \mathrm{FFA}}_2,{\mathrm{Real} \mathrm{FFA}}_3)).$$

(5)

2.4. Model Training and Testing

Our model was trained using registered and randomly cropped image groups with a batch size of 1 for 2000 epochs. The learning rate is 0.0002 in the first 1000 epochs and linearly decreased to 0 in the last 1000 epochs. The trained model is then used to translate fundus structure image to three-phase FFA images which are randomly cropped to generate final testing results with different field of views.

3. Results

A total of 7.3 h was required for training and 61.2 s for testing. These processes were performed on a computer with 64-bit Python, using the Ubuntu operating system with an Intel Xeon E5-2620 processor (2.10 GHz), 32 GB of memory, and an NVIDIA GeForce GTX 1080Ti graphics card.

Figure 6 depicts four typical examples of improvements achieved by our algorithm. The first and second examples are from a wide field of view, while the third and fourth are from a narrow field of view. Each example contains one input image, three output images obtained via real FFA imaging, three output images from our algorithm, three output images from Sequence GAN [19], three output images from pix2pix [22], and three output images from cycleGAN [23]. As shown in Figure 6, the FFA images translated by our algorithm show higher similarity with real FFA images than those translated by Sequence GAN [19], pix2pix [22], and cycleGAN [23]. As shown in the third phase images of the first and second examples, the FFA images translated by our algorithm, which are denoted by red square boxes, perform significantly better in the translation of optic nerve head (ONH) leakages. As a result, we qualitatively demonstrate our algorithm’s advancements.

To further validate our algorithm’s advancements, we compare FFA images translated by our algorithm to those translated by state-of-the-art algorithms [19,22,23] using PSNR, SSIM [24], and MSE. The PSNR, SSIM [24], and MSE values calculated using the entire final testing results are listed in

Table 1. Comparison of the similarity metrics for translated FFA: overall PSNR, SSIM [24], and MSE.

		PSNR ↑	SSIM ↑	MSE ↓
	Phase 1	16.5551	0.4361	2206.174
Our algorithm	Phase 2	15.8551	0.4536	2221.589
	Phase 3	16.2978	0.5338	2003.709
	Mean	16.236	0.4745	2143.824
	Phase 1	11.0153	0.3299	5809.015
Sequence GAN	Phase 2	12.8195	0.4081	3718.937
	Phase 3	13.4115	0.5466	3635.466
	Mean	12.4154	0.4282	4387.806
	Phase 1	13.2272	0.3162	4163.63
Pix2pix	Phase 2	14.0097	0.3882	2990.015
	Phase 3	14.5526	0.5313	3171.523
	Mean	13.9298	0.4119	3441.722
	Phase 1	10.0193	0.1691	7489.193
Cycle GAN	Phase 2	10.6604	0.2057	6165.381
	Phase 3	11.713	0.3957	5422.204
	Mean	10.7975	0.2568	6358.926

. Here, the real FFA images are considered as the ground truth. The PSNR and SSIM values are not very high due to the fluctuations of real three-phase FFA images and limited size of our dataset. Specifically, real three-phase FFA images are fluctuated by some measuring factors, for example, heart rates. Although the limited size of our dataset affects PSNR and SSIM values, these PSNR and SSIM values can also be used to validate our algorithm’s advancements. Based on these similarity indices, we identify two facts. First, our algorithm achieves higher mean PSNR and SSIM [24], and lower mean MSE, which indicates that the FFA images translated by our algorithm are overall more similar to real FFA images than those translated by Sequence GAN [19], pix2pix [22], and cycleGAN [23]. Second, the performances of our algorithm in the three phases are more balanced than those of Sequence GAN [19], which is caused by the unbalanced mutual information utilization rate of Sequence GAN [19]. Thus, the advancements of our algorithm are confirmed.

To demonstrate the performance of our method, two representative examples of our image translation are presented in Figure 7. The first example is from a wide field of view and the second example is from a narrow field of view. Each example contains one input image, three output images from real FFA imaging, and three output images from our algorithm.

As shown in Figure 7, the FFA images translated by our algorithm show high similarity with real FFA images. As shown in the third phase images of the first example, the FFA images translated by our algorithm, which is denoted by red square boxes, perform well in the translation of optic nerve head (ONH) leakages. Thus, it is shown that our method can provide good translated three-phase FFA images from fundus structure image.

4. Discussion

We presented a series-parallel GAN architecture for translating fundus structure image to fluorescence angiography. The proposed algorithm is aimed at translating fundus structure image to three-phase FFA images. Compared with the other algorithms [19,22,23], it achieves better performance due to its high utilization rate of the mutual information. This is achieved in two ways. First, it uses a series-parallel architecture-based generator that contains one encoder and three decoders. The encoder is used to extract the features from the inputted fundus structure image including mutual information of three-phase FFA images with the supervision of all the three-phase labeled FFA images. Three different weighted decoders are used to generate the three-phase FFA images, respectively. Second, it uses a single discriminator that takes the fundus structure image and three-phase FFA images as the input and then evaluates the FFA images based not only on single FFA image information but also on mutual information among the fundus structure image and three-phase FFA images.

Here, we make a comparison between our algorithm and other algorithms [19,22,23] in more detail. Compared with the three independent parallel GAN architecture [22,23], our algorithm has three advantages. First, our algorithm has lower computation costs for both the training and testing processes, due to the sharing of the encoder and the usage of a single discriminator. Second, our algorithm requires a smaller training set, because each translated FFA image is supervised by three-phase real FFA images at the same time. Third, instead of training each phase of the FFA image independently, our model is trained by the loss function that takes account of the fundus structure image and three-phase FFA images, which makes our training process more robust. Compared with Sequence GAN [19], our algorithm also has lower computation cost due to the sharing of the encoder and the usage of a single discriminator. In addition, our algorithm series-parallel GAN architecture improves the image translation quality of the first-phase FFA image by increasing its utilization rate of mutual information.

Because the translation from fundus structure image to multiple-phase FFA images is not yet available in commercial devices, there is still room for improvement. Here, we provide two options for future research. One option is finding a new GAN architecture to further improve the utilization rate of mutual information between the different phases of FFA images. The other option is translating fundus structure image to more than three phases of FFA images, which needs to capture more phases of FFA images and create the algorithm tailored for the datasets.

5. Conclusions

An automated algorithm for translating fundus structure image to three-phase FFA images is proposed. The algorithm comprises a series-parallel architecture-based generator and a single discriminator to achieve a high utilization rate of mutual information between the different phases of FFA images. The advancements of our algorithm over other state-of-the-art algorithms [19,22,23] are confirmed via qualitative and quantitative comparisons. Based on these comparisons, we conclude that this method shows superior performance in translating fundus structure image to three-phase FFA images by achieving higher similarity with real FFA images and better ONH leakages presentation, which may be useful for standard ophthalmic examinations as a screening diagnostic method in providing the information about the filling process of retinal vessels and detecting ONH leakages.