Deep Homography for License Plate Detection

Abstract

The orientation of plate images in license plate recognition is one of the factors that influence its accuracy. In particular, tilted plate images are harder to detect and recognize characters with than aligned ones. To this end, the rectification of plates in a preprocessing step is essential to improve their performance. We propose deep models to estimate four-corner coordinates of tilted plates. Since the predicted corners can then be used to rectify plate images, they can help improve plate recognition in plate recognition. The main contributions of this work are a set of open-structured hybrid networks to predict corner positions and a novel loss function that combines pixel-wise differences with position-wise errors, producing performance improvements. Regarding experiments using proprietary plate images, one of the proposed modes produces a 3.1% improvement over the established warping method.

1. Introduction

Automatic license plate recognition (ALPR) has had many real-world practical applications such as automatic toll collection, traffic law enforcement, parking lot access control, and road traffic monitoring. With increasing needs for intelligent transportation systems, ALPR has been studied actively despite its long history. In general, the pipeline of license plate recognition (LPR) systems has the four steps of image preprocessing, license plate detection, character segmentation, and optical character recognition (OCR). Various image conditions, such as perspective, color, font, occlusion, illumination, background, and country standards, make ALPR challenging.

ALPR is one of the active areas that are eager to benefit from the brilliant performance of deep learning methods. Such attempts began with using the Recurrent Neural Network (RNN) and regression boxes to localize texts under complicated environments [1]. Convolutional Neural Network (CNN) was the most successful solution for detecting plates typically coupled with sliding windows [2,3,4,5]. Variations of CNN such as Fast-RCNN and YOLO were proposed as well [6,7,8]. Recent approaches [9,10,11,12] have sought end-to-end architectures in which not only recognition but also image transformation occurs at the same time. Deep-learning-based approaches have been more efficient than traditional image processing methods, because feature decisions can be automated through the training process.

The outstanding performance of deep-learning-based ALPR is partly attributed to image-warping methods that rectify deformed plates in images. Even though image augmentation methods can help to improve the recognition accuracy and generalization of ALPR by artificially generating various types of deformed plates, warping methods have advantages over augmentation in terms of cost and computation. The warping task essentially involves matching and regression for parameters representing deformation, which can then be used for the rectification of plates, which leads to improved plate recognition.

In this paper, we propose deep learning models that are capable of predicting four-corner coordinates of deformed plate images, as shown in Figure 1. The four corners are then used to calculate transformation matrix, called homography, that rectifies plates. Whereas affine transformation is not always successful in rectifying plates because of its limitation that keeps parallel lines as they are, homography is general enough to handle various types of projective transformation. The homography is as follows in Equation (1).

$$w\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

(1)

where $\left(x,y\right)$ in one image is mapped to $\left(x^{\prime },y^{\prime }\right)$ in a target image. Since the homography matrix has the degree of freedom of 8 with $h_{33}=1$, we need a four pair of matching 2D coordinates to determine the matrix parameters of $h_{ij}$.

In this paper, we tackle the four-corner prediction problem by building and experimenting with a set of deep neural network models. We contribute by proposing an open neural network that can plug in well-known models such as Mobilenet and autoencoders in order to harness their power of feature extraction. Another contribution is a novel loss function that considers prediction errors not only in terms of coordinates but also in terms of image level. This paper is organized as follows. We review ALPR-related works in Section 2. We present our models in detail in Section 3 and discuss the experiment results in Section 4. We conclude the paper in Section 5 by discussing future work.

2. Related Work

Traditional methods for LPR have had the structure of the two steps involving text segmentation and following text recognition [13,14,15,16]. These methods, which depend mainly on the presence of discriminative characteristics such as color, shape, edge, or texture, have achieved limited success. The most notable disadvantage of the methods has been slow speed due to their inherent complex structure.

With the advent of deep learning methods, many research efforts have focused on a single-stage structure removing the segmentation step. Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) are able to extract features by scanning through sequential features of license plates [17]. A fully convolutional network implements segmentation and annotation-free ALPR by using synthetic plate images and plate character samples [18]. Most segmentation-free works depend on features that are learned by using Deep CNN enabled by residual layers [19].

However, these methods usually consider as input high-quality license plate images that have little variation in terms of illumination, plate obliqueness, and intensity. Such quality requirements often lead to low performance in actual field tests. To this end, there have been works that integrate image quality improvement in networks [11,12,20]. The main concern of their works is the rectification of slanted plates in images. A framework that includes de-noising and rectification is proposed for recognition improvement [20]. An affine transformation is estimated to align slanted plates [12]. A super-resolution technique is introduced to handle unconstrained real-world traffic scenes [11].

ALPR has been considered as a mature problem and misunderstood as easily solvable with deep learning. However, once the assumptions for input images are loosened, it poses a new level of challenges, typically in unconstrained scenes. In this work, we propose deep learning models that locate four-corner positions of plates, which are required to rectify plates. Unlike previous works that use affine transformation for rectification, our works build a homography that is a more versatile transformation.

3. Deep Models for Corner Prediction

We propose deep neural network models, as shown in Figure 2, which predict four-corner coordinates of license plates in an end-to-end manner. With the coordinates, we build a transformation matrix rectifying plates, resulting in improved character recognition accuracy in ALPR. The models share a similar two-phase structure; the first phase extracts geometric characteristics, which are then fed to fully connected layers of the second phase, which performs the prediction of the coordinates.

Figure 2a is a baseline model that consists of a set of convolution layers followed by a set of fully connected layers. The input is an image with dimension of W × H × C, and the output is a length-8 vector corresponding to four pairs of normalized $\left(x_i, y_i\right), 1\le i\le 4$. A hybrid model replacing the convolution layers with Mobilenet [21] is presented in Figure 2b. Its motivation is to avoid reinventing the wheel by having well-known object detection model extract geometric features. The next model of Figure 2c is the same as the hybrid model, except that the dropout layers are inserted between the fully connected layers for the model generalization. In the sense of feature extraction, it seems more appropriate to use the intermediate results of Mobilenet rather than the classification results. This observation motivates the configuration of Figure 2d that connects the output of the hidden layers of Mobilenet to the fully connected layers. The idea of Figure 2e stems from the fact that the encoders of autoencoder networks are good at capturing characteristics of images. It encourages one to feed the latent results from an autoencoder to the fully connected layers. We train the autoencoder in advance to be capable of reconstructing plates.

The models are open in the sense that, besides Mobilenet, other established networks can be plugged in the place of the feature extraction. A relatively small size but comparable performance is the main reason to use Mobilenet in our configuration, because one of the ALPR requirements is compactness for on-site installation. However, we do not limit the possibility to plug in other models such as Resnet [19] and MnasNet [22].

The second phase layers consist of fully connected layers for regression. They are different only by the dimensions of their inputs, which are dependent on the outputs from their corresponding previous layers. The regression output should be de-normalized before use by multiplying with image dimension.

As a loss function for an input image $I$, we use a weighted sum of two loss factors as Equation (2):

$$L\left(I\right)=\lambda ·L_{ssd}\left(I\right)+\left(1-\lambda \right)·\alpha ·L_{diff}\left(I\right)$$

(2)

where $L_{ssd}\left(I\right)$ is related to how far the predicted corner coordinates are from ground truth values and $L_{diff}\left(I\right)$ is about how different the rectified images based on the predictions are from ground truth image in pixel-wise. $\lambda$ determines the proportion of each factor with $\mathsf{\lambda }\in \left[0,1\right]$, and $\alpha$ is a constant that is needed to compensate the volume difference between two factors.

$L_{ssd}\left(I\right)$ is the sum of squared differences between four corresponding coordinate pairs of ground truth and predicted coordinates as Equation (3).

$$L_{ssd}\left(I\right)={\displaystyle \sum }_{i=1}^4{\left(x_i^{gt}-x_i^{pred}\right)}^2+{\left(y_i^{gt}-y_i^{pred}\right)}^2$$

(3)

where $\left(x_i^{gt},y_i^{gt}\right)$ and $\left(x_i^{pred},y_i^{pred}\right)$ are the normalized coordinates of ground truth and prediction, respectively, and $i$ represents the positions.

The other loss factor of $L_{diff}\left(I\right)$ is L1 loss for the transformed input image as Equation (4)

$$L_{diff}\left(I\right)=\frac{1}{N}{\displaystyle \sum }_{\left(i,j\right)\in N}^N{\Vert I_{i,j}^r-H{\left(I\right)}_{i,j}\Vert }^1$$

(4)

where $H{\left(I\right)}_{i,j}$ is the perspective transformation result of $I$ based on the prediction and $I_{i,j}^r$ is the ground truth of rectification that can be easily obtained from ground truth four-corner positions. Having two losses helps to preserve the appearance of plates such as intensity and illumination, and leads to denoising results capable of only geometric transformation.

Having four-corner predictions, the rectification begins by determining the target coordinates to which the predicted ones are mapped, which are the four corners of the plate rectangle without obliqueness. The target coordinates can be easily decided assuming the sizes and aspect ratio of plates are known in advance. Once one-to-one correspondence relationship between the target and the predictions is established, a homography matrix is determined by which actual rectification proceeds.

4. Experiments

We use a proprietary data set of 1137 plate images, all of which are tilted. All the images are preprocessed to be gray and have a size of 416 × 416. Since plates have wide width that is not able to fit the whole images, the bottom part remained empty. For training purposes, we manually labelled four corner coordinates of plates in separate files. The four corners are $p_i, i=1\dots 4$ of $\left(x_i, y_i\right)$; $p_1$ is the left top corner, and the rest are numbered clock-wise as in Figure 1. The coordinate $p_i$s is normalized as in YOLO [23] by dividing $x_i$ and $y_i$ by image dimension, having $0\le x_i,y_i\le 1$.

We split the data set by a ratio of 8:2, resulting in 909 images for the training set and 228 for the validation set. For training, we apply augmentation to the data set for model generalization and the diversification of plate obliqueness because most of the plate images are slanted to the right. The five types of augmentation are shown in Figure 3; they are symmetrical transformation, skewing, or the composition of them.

We train each of the models for 500 epochs on a same GPU, using an Adam optimizer with learning rate of 0.002 and weight decay of 0.0004. We use the batch size of 32. For the models including Mobilenet, we use the implementation of reference [24] with provided pretrained weights. Regarding the autoencoder, we train the autoencoder for 500 epochs with the data set and use only the encoder part. We implement each of the models using PyTorch [25], and source code is available at https://github.com/kyungkoo70/deep-plate-homography.

For performance comparison, we consider the WPOD-NET. However, it requires additional works before we can compare it with ours by the errors of predicted corners because the outputs of WPOD-NET are affine transformation that rectifies plates. Moreover, since WPOD-NET is tightly integrated into the end-to-end license plate recognition networks, it needs to be separated as an independent module. To calculate four-corner positions of WPOD-NET, we use an indirect way that applies the inverse matrix of the affine transformation to the four corners of rectified plates. The four corners are known because they can be calculated from ground truth corner positions. Even though it is not the perfect comparison because of errors induced during transformation, it is the best workaround to our knowledge.

Table 1. The average differences of normalized predicted coordinates from ground truth. The models trained with the loss function with different $\lambda$ were used.

Method	$\lambda$
	0.0	0.2	0.4	0.6	0.8	1.0
Baseline	2.34 × 10−2	2.04 × 10−2	1.43 × 10−2	1.44 × 10−2	1.45 × 10−2	1.46 × 10−2
Hybrid	2.22 × 10−2	1.95 × 10−2	1.27 × 10−2	1.25 × 10−2	1.27 × 10−2	1.29 × 10−2
Hybrid with dropout	2.56 × 10−2	2.34 × 10−2	1.56 × 10−2	1.56 × 10−2	1.60 × 10−2	1.67 × 10−2
Hybrid using extracted feature	2.30 × 10−2	2.20 × 10−2	1.30 × 10−2	1.30 × 10−2	1.40 × 10−2	1.46 × 10−2
Hybrid with encoded	3.84 × 10−2	3.00 × 10−2	1.84 × 10−2	1.84 × 10−2	2.56 × 10−2	2.84 × 10−2
WPOD-NET [12]	1.29 × 10−2

shows the average L2 distance of the predicted corners from ground truth depending on $\lambda$. Since normalized coordinates are used, the results are extremely small. We observe that the hybrid model achieves the best result, 1.25 × 10⁻² at $\lambda =0.6$ among the considered models. The hybrid outperforms WPOD-NET by 0.04 × 10⁻², which is 3.1% improvement. On other values of $\lambda$ as well, we find the hybrid is superior to other models except WPOD-NET. From the results of extreme $\lambda$ such as 0.0 and 1.0, we infer that the pixel-wise losses is not effective as the coordinate error based losses. However, the pixel-wise loss is complementary to the position-wise loss because it operates on the image level, exploiting nongeometric information such as intensity and illumination.

Figure 4 shows tilted plates in the first row and the corresponding rectification results by using the homography based on the four-corner predictions from the hybrid model in the second row. We find that the results are mostly consistent, proving that the predicted corner locations match with actual plate corners. We also observe that the blur of the plate boundaries does not affect the outcome, implicitly showing the robustness of the model against the boundary variation.

Figure 5 shows the sizes of the proposed models. We observe the hybrid models including Mobilenet are smaller than others except the hybrid-feature model. It is approximately 27 times bigger than the hybrid, the hybrid-dropout, and the hybrid-encoder. The baseline is approximately 22 times larger than the three small-sized models. Their huge sizes are attributed to the high dimensionality of the outputs of the convolution layers.

5. Conclusions and Future Works

We have introduced the deep models to estimate four corner coordinates of tilted plates. Since the predicted corners can then be used to rectify plate images, they can help improve plate recognition in ALPR. For experiments using proprietary plate images, the hybrid model brings a 3.1% improvement over the established warping method. The main contribution is a set of open-structured hybrid networks used to predict corner positions. As an additional contribution, we presented a new type of loss function that combines the pixel-wise differences with the position-wise errors, bringing performance improvements.

For future work, we want to experiment with our models with more diverse types of established detection models to observe performance variance. This would give us the opportunity to gain greater insight and understanding about the models and the interactions between them. Moreover, we intend to explore different loss factors that can enforce the relative position between the corner predictions.