Unsupervised Scene Image Text Segmentation Based on Improved CycleGAN

Abstract

Scene image text segmentation is an important task in computer vision, but the complexity and diversity of backgrounds make it challenging. All supervised image segmentation tasks require paired semantic label data to ensure the accuracy of segmentation, but semantic labels are often difficult to obtain. To solve this problem, we propose an unsupervised scene image text segmentation model based on the image style transfer model cyclic uniform Generation Adversarial network (CycleGAN), which is trained by partial unpaired label data. Text segmentation is achieved by converting a complex background to a simple background. Since the images generated by CycleGAN cannot retain the details of the text content, we also introduced the Atrous spatial Pyramid pool module (ASPP) to obtain the features of the text from multiple scales. The resulting image quality is improved. The proposed method is verified by experiments on a synthetic data set, the IIIT 5k word data set and the MACT data set, which effectively segments the text and preserves the details of the text content.

1. Introduction

In the last decade, scene text has been extensively studied. Since the development of convolutional neural networks, many solutions have been proposed. Scene text segmentation focuses on precisely separating the text from the background and outlining the character blocks. Therefore, text segmentation is an effective means to obtain text information in images, and it is a research hotspot in the computer field.

The difficulty of this work is that the font, color, and shape of the scene text are very different. The early methods based on image processing are mainly threshold algorithms [1,2,3,4,5,6], but they are very sensitive to the selection of threshold values in complex backgrounds, and the adaptability of different scenes is limited. Later, the proposed method combining text segmentation with probabilistic models [7,8,9,10] faced challenges such as data dependence and parameter estimation. Clustering methods [11,12,13] are also used to deal with text segmentation. With the development of neural networks, convolutional network-based methods can effectively handle various cases. However, the bottleneck of this approach is the high dependence on large-scale data, especially manually labeled data. For this reason, unsupervised segmentation networks have been proposed, such as fully convolutional adaptive networks (FCANs) [14], CyCADA [15] and CrDoCo [16]. However, these methods deal with the segmentation of street scenes, which is not helpful for the segmentation of text.

To this end, we propose using an unsupervised method to accomplish text segmentation with a complex background. We propose treating the segmentation task of scene text as a kind of style transfer of image, which is a novel perspective. We choose a CycleGAN network [17] that can carry out unsupervised style transfer. The two GAN networks are constrained by each other to form a cyclic adversarial network. A scene text image with a complex background is transformed into a text image with a simple background, and the model does not need to consider the interference of the complex background in the image. CycleGAN, on the other hand, learns network mapping without supervision. Scene image text segmentation is relatively simple in terms of semantic content, but the details of text strokes are more important. Therefore, in order to improve the edge detail of the generated image, we introduce the ASPP module [18]. When acquiring image features, ASPP can expand the receptive field, extract more abundant global information, improve the edge information of CycleGAN when generating segmentation images, and reduce the edge noise.

This paper makes the following contributions:

• This paper regards scene image text segmentation as a style transfer problem between images. From a scene text image with a complex background, after processing, it is transformed into a text image with a concise background, which provides convenience for subsequent processing. This transformation process actually reveals a novel perspective.
• This paper uses CycleGAN for the text segmentation of scene images. The detail processing of the edge of the text segmentation image generated by CycleGAN is not enough and not clear enough. The ASPP module is introduced into CycleGAN’s generator to obtain multi-scale text features and retain the details generated by image text edges.
• This paper conducts experiments on synthetic data sets, IIIT 5k word data sets, MACT data sets and MTWI data sets. Experimental results show that our method can effectively segment text while retaining clear edge information.

2. Related Work

2.1. Scene Text Segmentation

Scene text with complex background is the most challenging problem in scene text segmentation. Traditional text segmentation methods mainly include the threshold algorithm, probabilistic model algorithm and clustering algorithm. Each of these methods has its own characteristics and plays an important role in different application scenarios. As a key technology in text segmentation, the threshold algorithm is mainly used to distinguish the foreground (such as text) and background in an image. According to different threshold processing methods, it can be further divided into the global threshold method [1,2,3] and local threshold method [4,5,6]. The probabilistic model approach [7,8,9,10] has significant advantages in dealing with uncertainty, prediction and interpretation. The Gauss hybrid model and Markov random field model are representative of them. The clustering method [11,12,13] performs well in processing degraded text extraction in videos, mainly by mining the similarity between pixels to achieve the aggregation of text regions. Convolutional neural networks with learning ability can process all kinds of scene images, but their bottleneck is the high dependence on large-scale data, especially on manually labeled data. Bonechi et al. [19] generated COCO-TS and MLT-S. The COCO-Text [20] data set and MLT data set [21] were used for the weakly supervised processing of these two new data sets to obtain pixel-level annotations. However, the annotation quality of the new machine-generated data set COCO-TS is still far from that of manual annotation. Wang [22] proposed a network in which two branches guide each other. One branch generates the mask of the polygon, and the other branch generates the pixel-level text mask. The whole network training adopts a semi-supervised mode, but the mask is generated in the process of data processing. The accuracy of supervised segmentation methods depends on paired data, but the amount of paired data is limited, and manual labeling is time consuming. Therefore, in recent years, segmentation algorithms with unsupervised methods as the core have attracted much attention, such as fully convolutional adaptive networks (FCANs) [14], CyCADA [15], CrDoCo [16], etc. The experiment of transferring FCAN [14] from game video (GTA5) to an urban street scene has achieved good results, but the experimental effect of expanding to specific segmentation scenes needs to be verified. The CyCADA network [15] combines a cyclically consistent adversarial model of image translation with adversarial adaptive methods, primarily for urban streetscapes and digital. The unsupervised domain adaptive adversarial learning method of the CrDoCo network [16] also aims at the semantic segmentation of street scenes.

2.2. Image-to-Image Style Transfer

Image-to-image translation is used to transform the source domain into the target domain, which includes an image style transfer. The origin of this idea is the image analogy proposed by Jacobs et al. [23]. Recently, one-to-one image domain transformations have been scaled up. Isola [24] proposed the Pix2Pix architecture, which uses an end-to-end generative adversarial network as a framework. To preserve the underlying structure of the images, jump connections are used in the generators. However, the network is built with a one-to-one mapping of X-Y domains, which can lead to poor model diversity. Zhu et al. [25] designed BicycleGAN to address this problem by generating two different encodings with bidirectional affine to achieve diverse outputs. The pix2pixHD model [26] was proposed by Wang et al. It builds on Pix2Pix, which produces higher-resolution images. The multi-scale generator can also generate features from a fine-grained level. At present, the most advanced monitoring method is pix2pixHD. However, the supervised approach is complicated by the data it requires to pair, and semi-supervised approaches have received more attention. Zhu et al. proposed CycleGAN [17], which cleverly combines GAN networks with cycle-consistent loss. It uses unpaired data to accomplish the task of image style transfer. In this paper, we utilize CycleGAN to treat the problem of natural image text segmentation as an image-to-image style transfer.

3. Scene Text Segmentation Image Generation

3.1. Overview of the Architecture

CycleGAN [17] is a ring-shaped structure, as shown in Figure 1a, mainly consisting of two generators $G_A$, $G_B$ and two discriminators $D_A$, $D_B$. a denotes images in domain A, and b denotes images in domain B. There are two direction processes in CycleGAN. As shown in Figure 1b, the original image of domain A is transferred to domain B by the generator $G_A$, and then it is re-transferred to domain A by generator $G_B$. Figure 1c shows the other direction’s procession. In order to ensure the style transfer of the image, the discriminators $D_A$ and $D_B$ play a discriminating role.

We use CycleGAN to complete text segmentation of scene images. Images in domain A are scene text images with a complex background; images in domain B are text images without background and can be regarded as images with semantic labels. However, images in domains A and B are not paired. In this way, we convert the task about scene image text segmentation into a supervised image style transfer task.

In this paper, the generation process of the scene text segmentation network is shown in Figure 2. The scene text image a with a complex background is converted by the generator $G_A$ to $\widehat{b}$ with the background image removed, and then it is reconstructed back to a complex background image $\widehat{a}$ by the generator $G_B$. The discriminator $D_B$ is used to determine whether the generated image is a text image with the background removed. The image b without background generates image $\tilde{a}$ with complex background through the generator $G_B$, and then it restores it to the image $\tilde{b}$ without background through the generator $G_A$. The discriminator $D_A$ is used to determine whether generated images are complex background images. The text segmentation images generated using CycleGAN are less effective in terms of details, so we change the structure of the network to enhance the detail of the segmented images by introducing the ASPP module in the generator.

After model training is completed, we need to use the generator $G_A$ for text segmentation of the scene image. Inputting the scene text image in domain A into the generator $G_A$ will generate text image without background and complete scene text segmentation.

3.2. Network Architecture

3.2.1. ASPP

In the segmentation task, it is very important to keep the resolution of the image and the context information of different scales for the accurate segmentation of the task target. The Atrous convolution in the ASPP module [18] not only does not sacrifice the feature map resolution, it also expands the receptive field and captures multi-scale context information by using the Atrous convolution with different expansion rates in parallel.

Figure 3 shows the concrete structure. The components of ASPP include the following: a $1\times 1$ convolution, Pooling pyramid, and ASPP Pooling. ASPP can be extracted from multi-scale features, which is because we can customize the pooling expansion factor of each layer of the pyramid.

Atrous convolution has an expansion rate, which is different from the ordinary convolution. The expansion rate controls the padding and dilation during convolution. Through different padding and expansion, different scales of receptive field can be obtained, and multi-scale information can be extracted. It is noteworthy that the convolution kernel size is constant at $3\times 3$.

ASPP Pooling starts with an AdaptiveAvgPool2d layer. The so-called AdaptiveAvgPooling is adaptive in that it is unnecessary to specify the Kernel size and stride; only the final output size ($1\times 1$ in this case) needs to be specified. By compressing the feature maps of each channel separately to $1\times 1$, the features of each channel are thus extracted, and the global features are obtained. Then comes the convolution layer of $1\times 1$, which further extracts the features obtained in the previous step and reduces the dimensionality. It is worth noting that ASPP Pooling only extracts features. However, in addition to the sequential execution of the layers of the network, the feature map is finally sampled back to the original size from $1\times 1$.

3.2.2. Generator

CycleGAN includes two generators, both of which have the same structure and are composed of an encoder, a transformer, an ASPP module and a decoder. However, the generator $G_A$ realizes the conversion from a complex background image to no background image, $G_A:A\to B$; the complex background is removed. The generator $G_A$, on the other hand, restores the non-background image to a complex background image, $G_B:B\to A$. Figure 4 shows the internal structure of the generator in detail.

Both the input and output of the generator are RGB images. In order to reduce the computational effort, the input image is mapped from higher dimensions to lower dimensions. In other words, the input image is downsampled. Downsampling uses one convolutional layer of stride 1 and two convolutional layers of stride 2. In this way, the computational cost is the same, but you obtain a larger and more detailed network and receptive field. It can make the style transfer complete better. The converter consists of six residual blocks, which are responsible for extracting features in the image and converting the extracted image feature vector from domain A to domain B. After the feature vector conversion of the image is completed, it will enter the newly introduced ASPP module. The ASPP module continues to process these features and obtain global features so as to fuse features from multiple scales and improve segmentation details. The decoder reconstructs the output of the converter to the size of the original image. The process of image processing by the decoder is the inverse process of the encoder. It uses deconvolution layers to map images from low to high dimensions until the resulting image is equal in size to the original image.

When CycleGAN is used to complete the task of scene text segmentation, the text edge is not clear enough, and there is a lot of noise. We consider it because in the convolution process, the global information is ignored and the edge contour is not obvious. The ability of ASPP to capture global information is well known, so we chose to introduce ASPP into the generator.

3.2.3. Discriminator

The discriminator has a simple construction. Its role is to determine whether the input image is an image of a particular domain. As shown in Figure 5, the discriminator firstly uses four-layer convolution and an activation function to extract and process image features, and finally, it obtains a Patch output through convolution, taking the number of output channels as 1 to realize the purpose of “discrimination”.

In order to change the output into a single value of 0 or 1, the traditional GAN network discriminator usually uses the sigmoid function at the end, which is a judgment and evaluation of the discriminator on the input image, with 1 representing true and 0 representing false. However, the discriminator in CycleGAN adopts the idea of PatchGAN [17]. The so-called Patch means that after a series of convolution layers, the image will not be directly input into the full connection layer or a simple activation function. Instead, the use of a channel number 1 convolution will feature mapping for the $N\times N$ matrix. The function of this matrix is equivalent to the traditional single evaluation value. Compared with the discrimination mode of output only one value, the output $N\times N$ matrix has the advantage of the matrix of every point on behalf of the original image in the value of a small area, which is the meaning of Patch. Instead of using a single value to measure the whole picture, the whole picture is now evaluated using an $N\times N$ matrix. In this case, the label is also set to an $N\times N$ format, so that the loss calculation can be performed. The advantage of PatchGAN is that it uses more “receptive fields” and can combine more areas.

3.3. Loss Functions

3.3.1. Adversarial Loss

The image to image translation is actually the translation of domain A and domain B. For the CycleGAN model, the translation from the image in domain A to the image in domain B is achieved in the generator $G_A$ and then through the discriminator $D_B$ to determine whether the data are real. This is the process of generating antagonism in the GAN network, and the loss generated in the process is defined by the following formula:

$$L_{GAN}\left(G_A,D_B,A,B\right)=E_{b\sim P_{\mathrm{data}\left(b\right)}}\left[log{D}_B\left(b\right)\right]+E_{a\sim P_{\mathrm{data}\left(a\right)}}\left[log\left(1-D_B\left(G_A\left(a\right)\right)\right)\right]$$

(1)

The ultimate goal is shown below:

$$G_A^{*}=arg\underset{G_A}{min}\underset{D_B}{max}{L}_{GAN}\left(G_A,D_B,A,B\right)$$

(2)

Instead, another generator, $G_B$, will transform the image in domain B to the image in domain A. Judging by another discriminator $D_A$, the generated data are real. The losses resulting from this generative antagonism process are as follows:

$$L_{GAN}\left(G_B,D_A,B,A\right)=E_{a\sim P_{\mathrm{data}\left(a\right)}}\left[log{D}_A\left(a\right)\right]+E_{b\sim P_{\mathrm{data}\left(b\right)}}\left[log\left(1-D_A\left(G_B\left(b\right)\right)\right)\right]$$

(3)

The ultimate goal is shown below:

$$G_B^{*}=arg\underset{G_B}{min}\underset{D_A}{max}{L}_{GAN}\left(G_B,D_A,B,A\right)$$

(4)

3.3.2. Cycle-Consistent Loss

CycleGAN adds cycle-consistent loss to learn the mapping of $G_B$ and $G_A$. $G_A$ is used to transform the image a in domain A into domain B space and then restore it to a. Images in domain A may all be converted to the same image in domain B. To avoid this situation, calculate the loss between a and $G_B\left(G_A\left(a\right)\right)$. Similarly, all images in the B domain may be converted to the same image in the A domain. Again, the loss between b and $G_A\left(G_B\left(b\right)\right)$ needs to be calculated to avoid this. The cycle-consistent loss function was added to make the training process of CycleGAN more stable. Add restrictions to preserve the characteristics of the input image and reduce the mapping space. The loss function for cycle-consistent is defined as follows:

$$L_{cyc}\left(G_A,G_B\right)=E_{a\sim P_{\mathrm{data}\left(a\right)}}\left[{∥G_B\left(G_A\left(a\right)\right)-a∥}_1\right]+E_{b\sim P_{\mathrm{data}\left(b\right)}}\left[{∥G_A\left(G_B\left(b\right)\right)-b∥}_1\right]$$

(5)

3.3.3. Identity Loss

We use the generator $G_A$ to generate B-style images, so entering b into $G_A$ should also generate b. Therefore, $G_A\left(b\right)$ and b should be as similar as possible. Without this loss, the generator may automatically modify the tone of the image, resulting in a change in the overall color.

$$L_{Identity}\left(G_A,G_B\right)=E_{b\sim P_{\mathrm{data}\left(b\right)}}\left[{∥G_A\left(b\right)-b∥}_1\right]+E_{a\sim P_{\mathrm{data}\left(a\right)}}\left[{∥G_B\left(a\right)-a∥}_1\right]$$

(6)

3.3.4. Overall Loss

So, the total loss generated by the whole network training is shown below:

$$L_{total}=L_{GAN}\left(G_A,D_B,A,B\right)+L_{GAN}\left(G_B,D_A,B,A\right)+\alpha L_{cyc}\left(G_A,G_B\right)+\beta L_{Identity}\left(G_A,G_B\right)$$

(7)

$\alpha$ is the weight of the cycle-consistent loss and $\beta$ is the weight of the identity loss.

4. Experiments

4.1. Experimental Setting

4.1.1. Data Preparation

The training set and testing set used in the experiment are generated by Text-Recognit ion-Data-Generator [27]. Text-Recognition-Data-Generator is an open-source OCR data set generation project, which can customize scene text images according to the specific scene data set. We synthesized 26,352 images using 10,000 images as backgrounds and the text set in the project as foregrounds, including 13,143 images in the domain A (complex background text images) and 13,209 images in the domain B (no background text images). The training set consists of 10,143 A-domain images and 10,009 B-domain images. The test set consists of 3000 A-domain images and 3200 B-domain images.

4.1.2. Experimental Parameter Settings

The experiments were performed on an Ascend910 with the MindSpore framework. The input and output of generators $G_A$ and $G_B$ are both $256\times 256$ RGB images; the encoder in the generator includes a $7\times 7$ convolution with a step size of 1 and two $3\times 3$ convolutions with a step size of 2, and the figure of channels in the output is 64, 128, and 256, respectively; the residual block is designed as two $3\times 3$ convolution layers, and the structure is constructed based on the jump structure. The ASPP inflation factors are set to 6, 12, and 18; the decoder is set symmetrically with the encoder. The discriminators $D_A$ and $D_B$ adopt the design of patchGAN. We set the batch size to 1 and train the network for 100 rounds. The learning rate of the first 50 rounds is set to 0.0002, and the learning rate of the last 50 rounds decreases linearly until the attenuation is 0. The Adam optimizer is used for network optimization throughout the process.

4.1.3. Evaluation Criteria

In the evaluation of experimental results, this paper adopts two evaluation methods: qualitative and quantitative. The qualitative evaluation mainly focuses on the visual performance of the images generated by the model, such as the thoroughness of background removal, the integrity of text retention, and the overall natural degree of the image. Through visual observation of the generated images, we can preliminarily judge whether the model can effectively extract text content from the complex background. Quantitative evaluation is more objective and quantifies the performance of the model by calculating a series of evaluation indicators, which can more accurately reflect the ability of the model to remove background and retain text. By combining the qualitative and quantitative evaluation results, we can comprehensively evaluate the performance of the improved CycleGAN network model in processing text images with complex backgrounds.

There are two criteria for quantitative evaluation, which are described in detail below.

IS, inception score, is a common evaluation index in GAN. Its design is mainly based on two evaluation indicators of GAN: one is the quality of the result, that is, the clarity of the generated image; the other is the diversity of the generated image.

FID, Fréchet Inception Distance (FID), is also used to evaluate GANs, which was proposed by Heusel et al. It is used to evaluate the quality of images generated by GAN networks because it correlates with how well humans judge visual quality. The closer the distance between the data distribution of the generated image and the real image, the lower the value of the FID will be. We compute the FID between the test set of the data set and the relevant target domain image using the following formula:

$$FID(P_r,P_g)=∥{\mu }_r-{\mu }_g∥+T_r(C_r+C_g-2{\left(C_r{C}_g\right)}^{\frac{1}{2}})$$

(8)

The sample mean of the two data sets for FID calculation is expressed by ${\mu }_r$ and ${\mu }_g$; the sample covariance is expressed by $C_r$ and $C_g$.

IS is quantitatively assessed, and the calculated results are positively correlated with image quality. The better the quality of the generated image, the higher the calculated IS. The FID is unconstrained, and its calculation results are inversely proportional to the image quality. The lower the value of the calculated result, i.e., the closer the distribution of the generated image and the original image, the more visual properties are shared.

4.2. Experimental Results

A CycleGAN generator based on the codec structure is proposed, including an encoder, style converter and decoder. UNet is a typical codec structure with good segmentation performance. The difference of UNet is that UNet adds a skip layer structure SKIP-CONNECTION, which stitches together the feature maps of the same size before and after the codec, while the codec structure of ordinary networks is sampled to low latitude and then sampled to the original resolution. This method can preserve pixel-level details at different resolutions to a certain extent. Therefore, we changed the generator to UNet and tested it against CycleGAN’s original generator. At the same time, the ASPP module is referenced in the two generators respectively for ablation experiments. Experimental data and detailed experimental Settings are described above.

4.2.1. Qualitative Evaluation

The two generators are applied to the text segmentation of scene images with complex backgrounds, and the results of the experiment are compared. The qualitative results are shown in Figure 6, where we can visually compare the generated images of different networks. The first line is the input picture, and the second line is the segmentation results of the UNet as a generator. By observation, it is found that the segmentation effect is not good, and the network cannot distinguish between foreground and background. The third line shows the generation results after the introduction of the ASPP module for UNet, and the generation effect is not greatly improved. We believe that our task is to deal with text segmentation in complex backgrounds, and the UNet network cannot distinguish the background and text well when dealing with overly complex backgrounds. The fourth line is the original generator of CycleGAN, which can segment the complex background and foreground text well, but a lot of noise shadows appear around the text. The fifth line is the original generator with the introduction of the ASPP module, which is our proposed approach, and the noise shadows around the text are observed to be significantly improved.

In terms of qualitative evaluation, we believe that the original generator of CycleGAN is able to handle the segmentation of text in complex backgrounds well after the introduction of the ASPP module. The introduction of ASPP improves the overall effect of image generation and improves the details of image generation.

4.2.2. Quantitative Evaluation

We quantitatively evaluate the generated results through the evaluation indicators IS and FID. The value of IS is positively correlated with the quality of the image. As shown in

Table 1. Quantitative results.

	IS	FID
UNet	0.9985	248.1416
UNet+ASPP	2.3317	277.6030
The original generator	1.9735	176.2462
Ours	2.8725	155.7249

, the generation results of the two generators are compared. In our task, the original generator of CycleGAN has the highest IS value after the introduction of the ASPP module. The value of FID is negatively correlated with image quality. The results in the table show that the original generator of CycleGAN has the lowest FID value after the introduction of the ASPP module. It is noteworthy that with the introduction of the ASPP module, the value of IS increases by 45.6% and the value of FID decreases by 11.6% compared to the original generator. In other words, it improves the quality of the generated images to a great extent.

Combining the qualitative and quantitative results, the original generator with the ASPP module performs better in the task of text segmentation under complex background. At the same time, more text features and better visual effects are retained, which proves the validity of our experiment.

4.3. Experimental Effects on Other Data Sets

4.3.1. Data Preparation and Training Details

After determining the specific modules and architecture, we conduct identical experiments on The IIIT 5K-word data set [28] and MACT dataset [29].

The IIIT 5K-word data set [28] is obtained from Google Maps. The data set consists of 5000 word images, which are all cropped from Scene Texts and born-digital images. Of these 5000 images, 4000 are for the A-domain of the training set, and the remaining 1000 are for the A-domain of the test set. Overall, 5100 composite images are used in domain B of the training set, and 1140 are used in domain B of the test set.

MACT [29] is a multi-scene ancient Chinese text data set consisting of natural scene text in handwriting, calligraphy, and ancient fonts established by Kaili Wang et al. It contains 138,935 synthetic images for training and 14,318 real scene images for testing. We screen 13,000 pieces from the synthetic sample as the domain A of the training set and 1400 pieces from the real sample as the domain A of the test set. For the background-free text images in domain B, we collected a data set of background-free Chinese text images, including 5393 images of Wang Xizhi font, 5405 images of Yan Zhenqing font, and 2665 images of Li Shutong (Hongyi) font, with a total of 13,463 images for the training set domain B, and 1588 images of Zhu Da (Badashanren font) for the test set domain B.

The detailed numbers of each data set are shown in

Table 2. Numbers of training and test sets of the data set for the experiments.

	Training Set		Test Set
	Domain X	Domain Y	Domain X	Domain Y
The IIIT 5K-word	4000	5100	1000	1140
MACT	13,000	13,463	1400	1588

.

The experiments are conducted on Ascend910 with the MindSpore framework. In order to facilitate training and evaluation, all data are unified in the $256\times 256$ pixel size. Set the batch size to 1 and train the network for 100 rounds. The learning rate of the first 50 rounds is set to 0.0002; the learning rate of the last 50 rounds decreases linearly until the attenuation is 0. The Adam optimizer is used for network optimization throughout the process.

4.3.2. Qualitative Evaluation

(1) The IIIT 5K-word data set

The experimental results of the IIIT 5K-word data set are shown in Figure 7. Through observation, we find that the U-Net network as a generator is not good for segmenting the background and text in the scene image, and there is no significant improvement after the introduction of the ASPP module. The original generator of CycleGAN can segment the background and text. However, the segmentation effect is poor, and many noise points appear around the font, which affects the segmentation effect. After the introduction of the ASPP module into the original generator, the noise around the font is effectively handled, and the segmentation of background and text meets our expectation.

(2) MACT data set

The experimental results of the MACT data set are shown in Figure 8. By observation, we find that as a generator, the U-Net network does not achieve our expected results, and the segmentation effect is poor even if the ASPP module is added. The original generator of CycleGAN can segment the background and text better. However, blurred or redundant strokes will appear in the detail part, which is not allowed for Chinese recognition. After the introduction of the ASPP module, the blurred parts turn clear, and the redundant strokes are also removed.

In terms of qualitative evaluation, we believe that the original generator of CycleGAN has good segmentation effects on both the IIIT 5K-word data set and the MACT data set with the addition of the ASPP module. Moreover, the generated images perform well in terms of details and overall effects.

4.3.3. Quantitative Evaluation

The quantitative evaluations of the two data sets are shown in

Table 3. Quantitative results of different data sets.

	The IIIT 5K-Word		MACT
	IS	FID	IS	FID
UNet	3.0557	191.0428	1.7400	208.5827
UNet+ASPP	3.9968	198.7815	1.8932	192.9745
The original generator	2.7973	247.5222	1.9537	99.8899
Ours	4.1251	129.5635	2.0235	89.1945

. The information in the table shows that both U-Net as a generator and the original generator have improved the value of IS after the introduction of ASPP module. However, it is the original generator that has the highest IS value after the introduction of the ASPP module. For the FID value, both data sets have the smallest value for the original generator after the introduction of the ASPP module.

Combining the qualitative and quantitative results, the original generator with the introduction of the ASPP module will have good results for text segmentation in complex backgrounds with clearer segmentation details. This indicates that our experiment has universal applicability for scene image text segmentation tasks.

4.3.4. Synthetic to Real Adaptation

We visualized the image between a text image with a complex background and a text image without a background in Figure 9. Figure 9a is from the MTWI data set, which covers dozens of different font styles, font sizes ranging from a few pixels to a few hundred pixels, multiple layout layouts, and a variety of challenging interference backgrounds. These images are all from the real Internet environment. Figure 9c Text images without background from the composite data set, and their image spaces are transformed into opposite domains Figure 9b,d. Our model achieves a high practical domain transformation. This can explain the adaptability between image spaces. The most obvious difference between the original image and the generated image is whether there is a complex background. A text image without a background highlights the text information in the graphics, which is very important to us and is our goal.

The results in columns Figure 9a,b also show our adaptation from the composite image to the real image. Our method can complete the conversion from a text image with a complex background to a text image without background.

5. Conclusions

In this paper, the text segmentation of scene images is transformed into an unsupervised image-to-image style conversion task. We use the style transfer model CycleGAN to segment the text in the image. This eliminates the need to manually label data and pair data sets. We introduce the ASPP module into the generator to obtain multi-scale information features through convolution so as to refine the details of the generated segmented images. Our experiments were carried out on synthetic data sets, IIIT 5K-word data sets and MACT data sets. The improved CycleGAN network can significantly improve the processing of text edge information with the IS value increased by 45.6%, 47.5% and 3.57% and the FID value decreased by 11.6%, 47.7% and 10.7%, respectively. This validates not only the generality of our experiment but also that the introduction of ASPP can indeed improve the details of scene text segmentation However, the scalability and real-time performance of our approach is not outstanding, and we will continue to study it in the future.