CommuSpotter: Scene Text Spotting with Multi-Task Communication

Abstract

Scene text spotting is a challenging multi-task modulation for locating and recognizing texts in complex scenes. Existing end-to-end text spotters generally adopt sequentially decoupled multi-tasks, consisting of text detection and text recognition modules. Although customized modules are designed to connect the tasks closely, there is no interaction among multiple tasks, resulting in compatible information loss for the overall text spotting. Moreover, the independent and sequential modulation is unidirectional, accumulating errors from early to later tasks. In this paper, we propose CommuSpotter, which enhances multi-task communication by explicitly and concurrently sharing compatible information in overall scene text spotting. To address task-specific inconsistencies, we propose a Conversation Mechanism (CM) to extract and exchange expertise in each specific task with others. Specifically, the detection task is rectified by the text recognition task to filter out duplicated results and false positives, while the text recognition task is corrected by the rectified text detection task to replenish missing characters and decrease non-text interruptions. Consequently, the communication compensates for interaction information and breaks the sequential pipeline of error propagation. In addition, we adopt text semantic segmentation in the text recognition task, which reduces the complex design of customized modules and corresponding extra annotations. Compared with state-of-the-art methods, experimental results show that our method achieves competitive results with computation efficiency.

1. Introduction

Scene text spotting aims to detect and recognize various texts in complex scenes simultaneously. Derived from general object detection, scene text detection faces many challenges, such as scattered and changing characters in different word detection, text style and font variations, background interruptions, etc. On the other hand, scene text recognition also incorporates language processing techniques to decode sequence features for text prediction. The end-to-end training and application of scene text detection and recognition have garnered increased research interest in recent years. It not only integrates the two main tasks of text detection and recognition, but it also bridges the training schemas of different tasks for optimization. It has wide applications in artificial intelligence and computer vision, such as criminal investigation [1], video information retrieval [2], robotic assistance [3], and autonomous driving [4]. Modern scene text spotting comes up with challenges, such as artificially embellished texts, various text perspectives, and complex text shapes.

Existing two-stage paradigm text spotters conduct text detection tasks for text localization and text recognition tasks for words, while one-stage paradigm text spotters remove text detection but introduce customized modules for connecting the backbone with text recognition modules. The configuration examples of typical text spotters are collected in Table 1. Transformer-based backbones and text detectors greatly simplify proposal generation and boost performance at the expense of the computation cost. The customized modules are specifically designed to shrink the text instances or refine the features.

Although current end-to-end scene text spotters [5,6,7,8,9,10,11,12] have achieved substantial progress, there are three limitations. Firstly, the information or expertise from various multi-tasks compensates for each other in the final text spotting, but some expertise is lost in the current independent multi-task methods. For example, many studies improve information usage by sharing the network backbone [9,12,13], integrating the recognition loss [14,15], or bridging customized modules [5,6,7,8], as shown in Figure 1a. However, the interaction between tasks is not adequately identified. As indicated by the yellow arrow, there is only compensation from the text detection task (or customized modules) to the text recognition task, but not inversely, where the expertise from customized modules or text recognition is scarcely utilized to interact with early text detection tasks or proposal generations. There is no communication or usage of each other’s information for multi-tasks in the text-spotting process. Recently, some text spotters [14,15] have adopted a Transformer to understand the relationship between text instances but not between tasks. As a result, independent multi-task modulation leads to the loss of concurrent compensatory expertise from multiple tasks in text spotting. Secondly, apart from task-specific information loss, pipeline errors accumulate in current unidirectional text spotters. Errors from early tasks are not identified and, thus, accumulate in later tasks, leading to back-and-forth training. During inference, the methods become unreliable from the start when there is no ground truth in each step. Thirdly, the sophisticated customized modules increase model complexity and require expensive extra annotations. For example, some customized modules involve designs and labels for text lines [16], text strokes [17], text center points [18], and so on.

Table 1. Comparison between typical models. There are mainly two categories of methods divided by Transformer architecture. “CNN” is the traditional ResNet backbone. “Trans” stands for Transformer architecture. “Seg” is equal to “segmentation” for short. “Att” is the attention-based loss, while “CTC” is the CTC loss. Specific module techniques and details can be found in each method.

Category	Methods	Backbone	Proposals	Customized	Seg	Rec
CNN	Qin et al. [5]	CNN	RPN	RoI Masking	Instance	Att
	Mask TextSpotter v2 [19]	CNN	RPN	Box Detection	Instance	Att
	Mask TextSpotter v3 [6]	CNN	-	Seg Proposals	Instance	Att
	ABCNet [8]	CNN	-	Bezier Curve	Instance	CTC
	MANGO [7]	CNN	-	Mask Attention	Instance	Att
Trans	SwinTextSpotter [14]	CNN + Trans	Query Box	Recognition Conversion	Instance	Att
	TextTranSpotter [15]	CNN + Trans	Query Box	Hungarian Match	Instance	Att

Table 1. Comparison between typical models. There are mainly two categories of methods divided by Transformer architecture. “CNN” is the traditional ResNet backbone. “Trans” stands for Transformer architecture. “Seg” is equal to “segmentation” for short. “Att” is the attention-based loss, while “CTC” is the CTC loss. Specific module techniques and details can be found in each method.

Category	Methods	Backbone	Proposals	Customized	Seg	Rec
CNN	Qin et al. [5]	CNN	RPN	RoI Masking	Instance	Att
	Mask TextSpotter v2 [19]	CNN	RPN	Box Detection	Instance	Att
	Mask TextSpotter v3 [6]	CNN	-	Seg Proposals	Instance	Att
	ABCNet [8]	CNN	-	Bezier Curve	Instance	CTC
	MANGO [7]	CNN	-	Mask Attention	Instance	Att
Trans	SwinTextSpotter [14]	CNN + Trans	Query Box	Recognition Conversion	Instance	Att
	TextTranSpotter [15]	CNN + Trans	Query Box	Hungarian Match	Instance	Att

In this work, we propose CommuSpotter, which explicitly and concurrently communicates multi-task expertise with others. Multi-task communication involves information from all tasks complementing and assisting each individual task. Drawing inspiration from the image retrieval task [11], we develop a Conversation Mechanism (CM) to embed expertise vectors for multi-task interaction. Thus, this communication of expertise facilitates information exchanges among text detection and recognition modules, as shown in Figure 1b, breaking the unidirectional pipeline and reducing accumulated errors from the beginning. Additionally, we introduce text semantic segmentation for text recognition without requiring further annotations. This is achieved by adopting the independently pre-trained module weights as priors, replacing complex customized modules and their extra annotations. Finally, CommuSpotter constructs a concise framework and achieves fast convergence. The contributions of this paper are threefold: (1) Instead of independently and sequentially performing text detection (or customized modules) and text recognition multi-tasks, we developed CommuSpotter, which facilitates explicit communication through the designed Conversation Mechanism (CM). This mechanism embeds task-specific expertise concurrently across all tasks, compensating for task-specific information or expertise and reducing error propagation throughout the entire text-spotting process. (2) We employ text semantic segmentation expertise for text recognition tasks, reducing the need for complex custom module designs and their associated costly annotations. (3) We conduct comprehensive experiments on multiple text datasets. The comparisons between existing approaches demonstrate the advantages of our proposed method.

2. Related Work

2.1. Scene Text Spotter of Two-Stage Paradigm

To address the challenges of arbitrary texts, Lyu et al. [13] developed Mask TextSpotter v1, which includes long short-term memory (LSTM) [20] in text recognition tasks to boost spotting results. In Mask TextSpotter v2 [19], text and character instance segmentations are adopted to improve recognition performance. Qiao et al. [21] predicted additional latent information from text detection tasks to enhance text instance segmentation. Qin et al. [5] developed Regions of Interest (RoIs) masking to improve segmentation accuracy by selecting and fusing features for instances. These studies focused on improving the text recognition task. Some of the following focus on improving the early text detection task. FOTS [9] and TextNet [22] adopt rotating RoIs and perspective RoIs to handle irregular and multi-oriented text detections, respectively. CRAFT [23] groups character region features from the text detector to reinforce character attention for the recognizer. Kittenplon et al. [15] adopted Transformer to improve the representation from the shared backbone, while text detection and recognition tasks were parallel and independent. SwinTextSpotter [14] adopts the Swin Transformer [24] in the backbone for better text representations. The expertise from the text detection task is transferred to the text recognition task, but there is no inverse interaction. In addition, errors accumulate from early tasks to later tasks. Furthermore, some dataset annotations are tailored to specific methods, such as character and polygon annotations [13,25], text-line annotations [16], etc.

2.2. Scene Text Spotter of One-Stage Paradigm

Many customized modules have been developed to replace traditional text detection with bounding boxes, thereby building a closer connection between multiple tasks. For some segmentation-based methods, Mask TextSpotter v3 [6] uses segmentation proposals for arbitrarily shaped text recognition. Liu et al. [8,26] designed Bezier curves to represent text instances. MANGO [7] extracts text center lines and text and character segmentation maps for text grouping and recognition. For some regression-based methods, Wang et al. [27] designed the instance boundary points to improve text instance shapes. TextDragon [28] combines sliding RoIs with local points. SRSTS [29] adopts anchor points in text recognition. These methods replace bounding boxes with accurate text boundaries for text recognition, but customized modules are also little rectified by later expertise in text recognition, similar to two-stage paradigms. Again, the errors are propagated. Customized modules also require extra annotations, such as character annotations [6], stroke annotations [17], and so on.

2.3. Scene Text Spotter with Back-Propagation

Some studies show backward compensation among tasks but only build conversion by recognition loss. Zhong et al. [30] adopted a spatial transform network (STN) to propagate the recognition loss back to the text detection task. SwinTextSpotter [14] proposes the synergy mechanism for joint optimization by backpropagating the recognition loss. However, the loss function does not facilitate concurrent expertise interaction among sequential tasks, and it is not effective during testing in encoding representations.

3. Text Spotter with Communication

Scene text spotting mainly consists of text detection and recognition tasks, as shown in Figure 1. For the text detection task, the image is fed into a CNN-based network to extract feature maps, which are then pooled by $MaxPooling$ and $Softmax$ layers for the coordinate prediction of text locations and classification of text detection. The metrics include accuracy, recall, and F-score of text classification, as well as location overlap with the ground truths. Due to challenges in scene text variations, detection often contains many false positives, leading to incorrect recognition based on those detections. Text recognition relies on RNN networks that sequentially process the feature maps to predict each word character in text instances. For a maximum length of 25 for each word, each character is predicted and grouped to form the final recognition results. The metrics include recognition accuracy, recall, and F-score for every word.

3.1. Architecture

The whole framework of CommuSpotter is shown in Figure 2. It contains text detection and text recognition modules. Given an image, I, a backbone of ResNet-50 is used to extract feature maps, P, denoted as $\{P_2,P_3,P_4,P_5,P_6\}$, with a Feature Pyramid Network (FPN) following [31]. The Region Proposal Network (RPN) generates some Region of Interest (RoI) proposals, R, in five scales of $\{{32}^2$, ${64}^2$, ${128}^2$, ${256}^2$, ${512}^2\}$ on the pyramid features. The redundant candidates are filtered out by Non-Maximum Suppression (NMS). The pyramid features, P, and proposals, R, are processed by the RoIAlign [32] layer to generate the RoI features, $F_{RoI}$ and ${F^{\prime }}_{RoI}$, which are then fed into the text detection module and text recognition modules. Moreover, the semantic expertise, S and $S^{\prime }$, from the feature maps are extracted via softmax operations and attention-based refinements.

In the text detection module, the RoI features, $F_{RoI}$, interact with the expertise from the text recognition module through the specially designed Conversation Mechanism (CM) (in Section 3.2) to generate meaningful text detection through the classification layer. In the text recognition module, we adopt a segmentation-based method for text instance segmentation as the word expertise, as described in Section 2.2). Additionally, we introduce text semantic segmentation expertise S and $S^{\prime }$ as character details for text recognition. The representation is communicated by our designed Conversation Mechanism (CM) (in Section 3.3). Finally, text sequences are recognized by the RNN mechanism [33]. Note that the contents in this Section 3.1 are existing pipelines of scene text spotting, Section 3.2 and Section 3.3 below are our proposed approaches.

3.2. Text Detection Communication

Generally, the modality interaction [34] is either conducted by concatenating features and interacting together or by directly conducting interactions among features. Different tasks generate information from different perspectives for the final text spotting in text spotters. Existing text detection and recognition modules share the same backbone for different tasks. Considering the modeling efficiency, we keep the same backbone but introduce Extensive Representation (ER) on pyramid features for different tasks, as shown in Figure 3a. Prior to applying RoIAlign, as detailed in [32], which builds local coherence for feature alignments, we embed high-level pyramid features on a large receptive field for text detection. Specifically, the shared pyramid features $P_2$ to $P_5$ are updated as follows:

$$P_{i-1}^{*}=Gcm(Ur\left(P_i\right)·P_{i-1}),i=3,4,5,$$

(1)

where $P_2$ is the shape of $C\times H/4\times W/4$, $P_3$ is the shape of $C\times H/8\times W/8$, $P_4$ is the shape of $C\times H/16\times W/16$, and $P_5$ is the shape of $C\times H/32\times W/32$. C represents the dimension of channels, set at 256, while H and W denote the input resolutions in terms of height and width, respectively. The operation $Ur(·)$ is used for up-sampling. $Gcm(·)$ denotes a sequence of convolution layers, consisting of $1\times 3$ and $3\times 1$ convolutions. They are applied to reduce dimensionality gradually. Finally, the results are combined as $P^{*}$, consisting of $P_2^{*}$ to $P_5^{*}$.

Compared with the general object detection of integrated targets, split characters inside texts always cause false positive detection results due to background interruptions and character-like non-texts. To achieve concurrent semantic guidance, we developed a Conversation Mechanism (CM) through a series of straightforward operations. This mechanism incorporates the expertise of a text recognition module, comprising (1) text semantic segmentation expertise S (as detailed in Section 3.3) to identify character objects (other than non-texts) and (2) text instance segmentation expertise $F^{\prime }$ to construct exact locations. Specifically, the refined features, $P^{*}$, and proposals, R, are fed into the RoIAlign with text semantic segmentation expertise S to generate the aligned RoI feature $F_{RoI}$ of the shape $N\times C\times H_{RoI}\times W_{RoI}$, where N is the number of mapped proposals, and $H_{RoI}$ and $W_{RoI}$ are feature map resolutions set to 7. To obtain text instance segmentation expertise $F^{\prime }$, we conducted a sequence of four convolutional layers $Conv$ and one transposed convolutional layer $TrConv$ on the RoI features $F_{RoI}$. Then, the expertise was combined.

$$\begin{array}{c}{F}_{RoI}=RoIAlign(P^{*},R)+S,\end{array}$$

(2)

$$\begin{array}{c}{F}^{\prime }=TrConv\left(Conv\left(F_{RoI}\right)\right),\end{array}$$

(3)

$$\begin{array}{c}{F}_{RoI}^{*}=F^{\prime }+F_{RoI}.\end{array}$$

(4)

3.3. Text Recognition Communication

As mentioned above, a segmentation-based text recognition module adopts text or character instance segmentation. Similar to the text detection module, the shared features are first extended by the reverse Extensive Representation (ER) to embed global information, as shown in Figure 3b. Specifically, the filtered pyramid features, P, consisting of multiple levels, from $P_2$ to $P_5$, are presented for the module as follows:

$$P_{i+1}^{**}=Gcm(Dr\left(P_i\right)·P_{i+1}),i=2,3,4,$$

(5)

where $Dr(·)$ is the downsampling operation.

The instance segmentation-based recognition methods always suffer from the problem of missing characters in texts, especially when some characters inside a text are artistically designed with different textures, shapes, fonts, etc. From this view, the purpose of the Conversation Mechanism (CM) is to integrate (1) the renewed expertise from text detection module $F^{″}$ as global guidance and (2) text semantic segmentation expertise $S^{\prime }$, which provides local details. Specifically, the extensive features, $P^{**}$, and filtered proposals, $R^{\prime }$, from RPN are fed into the RoIAlign with text semantic segmentation expertise $S^{\prime }$, to generate the aligned RoI feature, $F_{RoI}^{\prime }$, of the shape $N^{\prime }\times C\times H_{RoI}^{\prime }\times W_{RoI}^{\prime }$, where $N^{\prime }$ is the number of instances, and $H_{RoI}^{\prime }$ and $W_{RoI}^{\prime }$ are the resolutions of 16 and 64, respectively.

$$\begin{array}{c}{F}_{RoI}^{\prime }=RoIAlign(P^{**},R^{\prime })+S^{\prime },\end{array}$$

(6)

$$\begin{array}{c}{F}^{″}=Lr\left(Lr\left(F_{RoI}^{\prime }\right)\right),\end{array}$$

(7)

$$\begin{array}{c}{F}_{RoI}^{**}=F^{″}+F_{RoI}^{\prime }.\end{array}$$

(8)

Then, the aligned RoI feature, $F_{RoI}^{\prime }$, is transferred by two consequent linear convolutional layers $Lr(·)$. Finally, we concatenate text detection expertise $F^{\prime \prime }$ with the RoI features $F_{RoI}^{\prime }$.

Another problem of segmentation-based recognition is that character details are ignored in text spotters. The pixel-level expertise of characters can provide information to differentiate characters of similar shapes. For example, “hot” and “hat” only differ in one character’s details. Due to the lack of pixel-level annotations for scene text datasets, text semantic segmentation is not applied in existing scene text spotters. Instead of designing complex modules to tighten the character regions with extra annotations, we introduce text semantic segmentation [35] expertise S and expertise $S^{\prime }$ with individually pre-trained weights. The initial semantic segmentation results, $S_e$, are refined by the attention mechanism [36] to obtain a better text semantic segmentation $S_{re}$, as shown in Figure 2.

$$\begin{array}{c}{S}_f=ResNet\left(I\right),\end{array}$$

(9)

$$\begin{array}{c}{S}_e=Softmax\left(S_f\right),\end{array}$$

(10)

$$\begin{array}{c}{S}_{att}=Att(S_e,S_f),\end{array}$$

(11)

$$\begin{array}{c}{S}_{re}=Convolution(S_e,S_{att}),\end{array}$$

(12)

where $S_f$ is the fusion of pyramid features $P_2$ to $P_5$, $Softmax$ is the softmax operation, $Att$ is an attention layer of the dot-product operation, and $Convolution$ is a series of convolution layers of kernel size 5 × 5 and 1 × 1, to fuse the attention with the feature maps. To communicate this expertise in the text spotter, we take pooling operations on text semantic segmentation $S_{re}$ with different resolutions. Then, text semantic segmentation expertise S and expertise $S^{\prime }$ are generated.

$$\begin{array}{c}S=\mathcal{P}\left(S_{re}\right),\end{array}$$

(13)

$$\begin{array}{c}{S}^{\prime }={\mathcal{P}}^{\prime }\left(S_{re}\right),\end{array}$$

(14)

where $\mathcal{P}$ and ${\mathcal{P}}^{\prime }$ are the pooling layers with different resolutions. The shape of expertise S is $N\times C\times H_{RoI}\times W_{RoI}$, and $S^{\prime }$ is $N^{\prime }\times C\times H_{RoI}^{\prime }\times W_{RoI}^{\prime }$.

3.4. Optimization

The whole framework loss $\mathcal{L}$ is defined as follows:

$$\begin{array}{c}\mathcal{L}={\mathcal{L}}_{rpn}+\alpha {\mathcal{L}}_{rcnn}+\beta {\mathcal{L}}_{rec},\end{array}$$

(15)

$$\begin{array}{c}{\mathcal{L}}_{rcnn}={\mathcal{L}}_{cls}+{\mathcal{L}}_{reg},\end{array}$$

(16)

$$\begin{array}{c}{\mathcal{L}}_{rec}={\mathcal{L}}_{ins}+\delta {\mathcal{L}}_{seg}+ϵ{\mathcal{L}}_{seq},\end{array}$$

(17)

where ${\mathcal{L}}_{rpn}$, ${\mathcal{L}}_{rcnn}$, and ${\mathcal{L}}_{rec}$ are the losses of RPN [13], text detection module, and text recognition module, respectively. The weights of $\alpha$ and $\beta$ are equal to 1.0 and 1.0, respectively. The detection module loss is ${\mathcal{L}}_{rcnn}$, consisting of the cross-entropy classification loss ${\mathcal{L}}_{cls}$ and the smooth L1 regression loss ${\mathcal{L}}_{reg}$ [37]. The text recognition module loss ${\mathcal{L}}_{rec}$ includes a cross-entropy text instance segmentation loss ${\mathcal{L}}_{ins}$, a character instance segmentation loss ${\mathcal{L}}_{seg}$, and a sequence recognition loss ${\mathcal{L}}_{seq}$ [13]. The weights of $\delta$ and $ϵ$ are empirically set to 1.0 and 0.2, respectively. The ${\mathcal{L}}_{seq}$ follows a summation of the logarithm loss [13]. The text instance map is encoded by convolutional and max pooling layers with two-dimensional representations [38], and fed into a seq2seq recognizer [39] to generate text sequences.

4. Experiments

4.1. Datasets

SynthText is a synthetic dataset [40] of around 800 k images with comprehensive text samples used for pre-training. ICDAR2013 (IC13) is for the 2013 Robust Reading Competition [41], consisting of 229 training images and 233 test images. ICDAR2015 (IC15) is provided by the 2015 Robust Reading Competition [42]. It contains incidental scene texts of 1000 training images and 500 test samples. Total-Text (TT) [43] focuses on arbitrarily shaped texts, including 1255 training and 300 test images. SCUT [44] contains 1162 natural images from Flickr [13]. These real data are used for fine-tuning the model. The evaluation is conducted on IC15 and TT, current scene text-spotting benchmarks.

4.2. Implementation Details

The model is trained using PyTorch with two Tesla-V100 GPUs and tested on a single GPU. Following the Mask TextSpotter v2 [19], the training process consists of two parts: pre-training and fine-tuning. The optimizer is Stochastic Gradient Descent (SGD), set with a weight decay equal to 0.001 and a momentum of 0.9. In the pre-training stage, the model is trained on the SynthText [40] for 270 K iterations. The initial learning rate is set to 0.01 and decayed at every 90 K iterations by a tenth. In the fine-tuning stage, the model is trained on multiple real-world image datasets for 90 K iterations. The learning rate is set to 0.001. In the inference stage, the input images are fed into the model to generate proposals, instances, and recognition predictions.

4.3. Ablation Study

The experiments adopt percentage values of the end-to-end recognition F-scores on the ICDAR15 dataset with a strong lexicon. We train a model that only adopts the Extensive Representation (ER) mechanism from the baseline Mask TextSpotter v2 [19]. The ER improves the recognition performance from 82.1% to 82.5% in

Table 2. End-to-end recognition results of ICDAR15 on different model configurations. “CM-P” stands for part of the Conversation Mechanism without text semantic segmentation expertise.

Methods	Extensive Representation	CM-P	Conversation Mechanism	F-Score
E2E-baseline				82.1
w/ER	✓			82.5
w/CM-P		✓		83.7
w/CM-P	✓	✓		84.6
Full Setting	✓	✓	✓	85.8

.

To validate the effectiveness of communication between text detection and instance segmentation tasks, we add the Conversation Mechanism (CM) to them but without expertise from text semantic segmentation. The recognition results indicate that concurrent communication can improve performance from 82.1% to 83.7%. If equipped with the above ER, it can achieve 84.6%. Finally, with full settings of CM, we add the interaction from the text semantic segmentation. The F-score is further improved from 84.6% to 85.8%. Thus, there is a collaborative effect for the entire text spotter.

We train and fine-tune several models of the configurations on the Total-Text dataset, as shown in

Table 3. End-to-end recognition results of Total-Text on different model configurations. “CM-P” stands for part of the Conversation Mechanism without text semantic segmentation expertise.

Methods	Extensive Representation	CM-P	Conversation Mechanism	F-Score
E2E-baseline				77.4
w/ER	✓			78.4
w/CM-P		✓		80.3
w/CM-P	✓	✓		81.7
Full Setting	✓	✓	✓	83.4

. The ER improves the spotting performance from the baseline Mask TextSpotter v2 [19] of 77.4% to 78.4%. The singular CM-P can improve the performance from 77.4% to 80.3%. Together with ER, the recognition results indicate that concurrent communication can improve performance to 81.7%. With full settings of CM, we add the interaction from text semantic segmentation. The F-score is further improved to 83.4%.

As mentioned above, some methods adopting Transformer or customized modules may train the model on extra datasets or make corresponding extra labels for the datasets. It causes difficulty in the fair comparison of different approaches. We compare the model efficiency in

Table 4. Comparison between training configuration and computation cost. In the pre-training, mix-training, and/or fine-tuning stages, the word “Data” represents the datasets used for training, and “Iter.” denotes the number of convergence iterations needed. “GPU” denotes the estimated GPU hours for each method.

Methods	Pre-Train		Mix-Train or/and Fine-Tune		GPU
	Data	Iter.	Data	Iter.
MANGO [7]	SynthText	600 K	CST, COCO, MLT, IC13, 1C15, Total	250 K	∼1344
ABCNet [8]	CST, COCO, MLT	260 K	Total, CTW	150 K	$\sim$288
SwinTextSpotter [14]	CST, MLT, IC13, IC15, TT	450 K	IC13, IC15, Total, MLT, CTW	90 K	∼336
TTS [15]	SynthText	-	SynthText, IC13, IC15, Total, SCUT, COCO	-	∼1200
Mask TextSpotter v2 [19]	SynthText	270 K	SynthText, IC13, IC15, Total, SCUT	150 K	∼432
Ours	SynthText	270 K	IC13, IC15, Total, SCUT	90 K	∼312

. For the abbreviation of datasets, “CST” is Curved SynthText [8]; “COCO” is COCO-Text [45]; “MLT” is ICDAR-MLT [46]; “IC13” is ICDAR2013 [41]; and “CTW” is SCUT-CTW1500 [47]. We compare the costs of computation resources with some approaches in Table 4. For the maximum iterations of convergence in the training process, we only need 360 K iterations in total, which is almost the fastest model. The GPU hour is roughly estimated from previous papers or re-implementation. Our GPU hour is a little more than the most lightweight ABCNet [8], but our performance is better than that work. As a result, our method can achieve competitive performance at a lower cost compared to previous approaches. It is a good trade-off between model performance and computational efficiency.

4.4. Incidental Texts

All the following comparison metrics on the text-spotting datasets are percentage values. We evaluate the method on incidental texts of IC15 [42]. The evaluation results are shown in

Table 5. Comparison results on the ICDAR2015 dataset. For the detection result, “P”, “R”, and “F” represent the metrics of precision, recall, and F-score, respectively. The end-to-end recognition evaluation is the F-score. “S”, “W”, and “G” means strong, weak, and generic lexicons, respectively.

Methods	Detection			End-to-End
	P	R	F	S	W	G
FOTS [9]	91.0	85.2	88.0	81.1	75.9	60.8
Qin et al. [5]	89.4	85.8	87.5	83.4	79.9	67.9
Mask TextSpotter v1 [13]	91.6	81.0	86.0	79.3	73.0	62.4
Text Perceptron [21]	91.6	81.8	86.4	80.5	76.6	65.1
TextDragon [28]	92.5	83.8	87.9	82.5	78.3	65.2
CharNet [25]	91.2	88.3	89.7	80.1	74.5	62.2
ABCNet v2 [26]	-	-	-	82.7	78.5	73.0
CRAFTS [23]	89.0	85.3	87.1	83.1	$\mathbf{82}.\mathbf{1}$	74.9
Boundary [27]	89.8	87.5	88.6	79.7	75.2	64.1
Mask TextSpotter v2 [19]	86.6	$87.3$	87.0	83.0	77.7	73.5
Mask TextSpotter v3 [6]	-	-	-	83.3	78.1	74.2
MANGO [7]	-	-	-	81.8	78.9	67.3
SwinTextSpotter [14]	-	-	-	83.9	77.3	70.5
TTS [15]	-	-	-	85.2	81.7	77.4
GLASS [48]	-	-	-	84.7	80.1	76.3
SRSTS [29]	96.1	82.0	88.4	85.6	81.7	74.5
Ours	91.4	88.3	89.8	85.8	80.2	74.9

. Our method achieves a state-of-the-art recall of 88.3% and an F-score of 89.8% compared with previous studies. For the end-to-end recognition evaluation, our method outperforms the previous non-Transformer methods in the strong and generic lexicon. The F-score achieves 85.8% and 74.9%. Compared with recent Transformer based methods, it is hard to distinguish the effects of the Transformer from the modulation. Also, TextTranSpotter (TTS) [15] achieves a higher F-score for the generic lexicon when trained with 43 K more images compared with our 4 K images. Our results demonstrate the effectiveness of communication among multi-tasks for end-to-end text spotting in incidental texts. Apart from the Transformer, this represents a promising exploration into the simple and clean modulation of text spotters.

4.5. Arbitrary Texts

To verify the effectiveness of the model on irregular texts, we conduct experiments on the Total-Text (TT) [40], following the detection and end-to-end evaluation protocols publicized with the dataset. The comparison results are presented in

Table 6. Comparison results on the Total-Text (TT) dataset. The detection results are measured by precision (P), recall (R), and F-score (F). The recognition F-scores include evaluation without lexicon as “None” and with lexicon as “Full”.

Methods	Detection			End-to-End
	P	R	F	None	Full
TextSnake [49]	82.7	74.5	78.4	-	-
Mask TextSpotter v1 [13]	69.0	55.0	61.3	52.9	71.8
TextNet [22]	68.2	59.5	63.5	54.0	-
Mask TextSpotter v2 [19]	81.8	75.4	78.5	65.3	77.4
Mask TextSpotter v3 [6]	-	-	-	71.2	78.4
Boundary [27]	85.2	83.5	84.3	65.0	76.1
CRAFTS [23]	89.5	85.4	87.4	78.7	-
ABCNet [8]	-	-	-	64.2	75.7
ABCNet v2 [26]	-	-	87.0	70.4	78.1
PAN++ [50]	-	-	-	68.6	78.6
MANGO [7]	-	-	-	71.7	82.6
SwinTextSpotter-R [14]	-	-	87.2	72.4	83.0
SwinTextSpotter [14]	-	-	88.0	74.3	84.1
GLASS [48]	-	-	-	76.6	83.0
SRSTS [29]	92.0	83.0	87.2	78.8	86.3
TTS [15]	-	-	-	75.6	84.4
Ours	90.4	91.5	90.1	73.0	83.4

. Our detection results achieve state-of-the-art performance compared with non-Transformer methods and achieve the best recall and F-scores compared to recent Transformer methods. The recognition results outperform previous non-Transformer text spotters while not as good as Transformer methods. However, the non-lexicon F-score of 73.0% is significantly improved from the baseline Mask TextSpotter v2 [19] of 65.3%. The recognition result of 83.4% with lexicon outperforms the corresponding baseline of 77.4%. Compared with SwinTextSpotter [14] of the ResNet backbone (shown as SwinTextSpotter-R), our recognition results achieve better performance. It shows that without a Transformer, our communication mechanism can be better in modulation. The newest SRSTS [29] achieves the best recognition results with Transformer decoders, which indicates that we still have improvement space in the recognition module.

4.6. Inference Speed

The inference speed is compared with previous approaches, as shown in

Table 7. Frames per second (FPS) comparison on different inference datasets.

Methods	ICDAR2015	Total-Text
Mask TextSpotter v2 [19]	3.1	-
Mask TextSpotter v3 [6]	2.5	-
ABCNet [8]	-	6.9
MANGO [7]	4.3	4.3
Ours	2.9	2.4

. Not all recent studies provide inference time statistics. Compared to MANGO [7], our method is a little slower but with better performance. As mentioned above, our method is more efficient at training and can be easily used in practical applications without complex modules and expensive data labeling.

4.7. Qualitative Results

We can see the improvements in our approach (second row) compared with the baseline Mask TextSpotter v2 [19] (first row) from Figure 4. For the first two examples with complex scenes, the original method always catches false positive detections and obtains wrong recognition results. That is, decorated curves are wrongly detected as texts and recognized as non-existing information. For the last two examples of arbitrarily shaped texts, the text shapes and background interruptions always cause incomplete text instances. For example, the curved character layout is easily spotted as duplicated text but not whole instances. As a result, there are many unclear recognized meanings in the scene. There are fewer errors in our method in the second row. We present more qualitative samples in the third and fourth rows of Figure 4 for comparison.

5. Conclusions

We proposed a streamlined framework to facilitate the exchange of expertise among multiple tasks in the scene text spotter. With the bidirectional communication between text detection and text recognition modules, our CommuSpotter allows for concurrent expertise exchange and early error correction. Instead of a complex, customized module design for tight character regions, we introduce text semantic segmentation in the recognition module. We conduct experiments on incidental and curved text datasets; the proposed method achieves consistently competitive performance with model efficiency.