A Comparative Study on Detection and Recognition of Nonuniform License Plates

Abstract

This paper presents a comparative study on license plate detection and recognition algorithms in unconstrained environments, which include varying illuminations, nonstandard plate templates, and different English language fonts. A prime objective of this study is to assess how well these models handle such challenges. These problems are common in developing countries like Pakistan where diverse license plates, styles, and abrupt changes in illuminations make license plates detection and recognition a challenging task. To analyze the license plate detection problem Faster-RCNN and end-to-end (E2E) methods are implemented. For the license plate recognition task, deep neural network and the CA-CenterNet-based methods are compared. Detailed simulations were performed on authors’ own collected dataset of Pakistani license plates, which contains substantially different multi-styled license plates. Our study concludes that, for the task of license plate detection, Faster-RCNN yields a detection accuracy of 98.35%, while the E2E method delivers 98.48% accuracy. Both detection algorithms yielded a mean detection accuracy of 98.41%. For license plate recognition task, the DNN-based method yielded a recognition accuracy of 98.90%, while the CA-CenterNet-based method delivered a high accuracy of 98.96%. In addition, a detailed computational complexity comparison on various image resolutions revealed that E2E and the CA-CenterNet are more efficient than their counterparts during detection and recognition tasks, respectively.

1. Introduction

Object detection and recognition is one of the important and challenging tasks in computer vision and understanding of the image [1]. These fields have drawn significant interest from researchers due to their diverse applications, which include parking management, human–computer interaction, vehicle tracking, autonomous driving, and industrial automation [2]. License plate (LP) detection and recognition is a more complex than a simple image classification task due to the fact that many times, such images contain multiple categories of these objects [3]. In real-world environments, the LPs can vary in size, shape, color, and condition and, therefore, make detection and recognition tasks time-consuming and intensive. High-speed vehicles, nonuniform plates, an environmental factors—for instance, smoke—and varying illumination considerably affect the LP detection and recognition. In countries like Pakistan, a huge percentage of on-road vehicles contain nonuniform LPs with varying sizes, colors, and fonts. Figure 1 shows a few such samples of the LPs that have different styles with different appearances and diverse language styles. The LP detection and recognition of such nonstandard LPs become more challenging due to various factors, such as shadows on the LP area, fancy LPs with embossed characters, and broken or mud-contaminated plates. In recent times, numerous machine learning-based LP detection and recognition algorithms have appeared in the literature [4].

Many of these methods use conventional machine learning, CNNs, and DNNs to effectively detect and recognize various object styles, for instance, vehicles and LPs with remarkable accuracy [5]. In addition to the above-mentioned issues, there are several other problems that significantly challenge the detection and recognition accuracy of state-of-the-art algorithms, for instance, low resolution, image quality, and complex background. Moreover, LP detection and recognition are an integral part of ITS and smart city planning. Deploying an accurate and real-time LP detection and recognition method is very crucial as it will add an extra layer to security and surveillance purposes by detecting and identifying LPs in real time. It will also assist in maintaining a proper traffic flow and identifying suspicious vehicles for law enforcement agencies to take timely action. An accurate LP detection and recognition system is also handy for public safety if properly used at various public gathering places, such as parking lots, borders, shopping malls, and toll plazas.

Existing LP detection and recognition methods use both traditional and deep learning-based approaches. The traditional LP handling methods depend on limited features and yield non-satisfactory performance for contour, color, and edge variations. On the other hand, deep learning-based methods learn a large number of features and have produced encouraging results. Therefore, inspired by the recent success in deep learning-based methods, our main contributions outlined in this manuscript are highlighted below.

• We present a comparative study of four algorithms, out of which two algorithms investigate LP detection, while the other two techniques analyze LP recognition performance in unconstrained traffic environments.
• We report experiments on a challenging Pakistani traffic dataset with nonuniform appearances. Compared algorithms are applied to a variety of different vehicles that have significant variations in the appearance of their LPs. In addition, the computational complexity of these methods has also been investigated on varying image resolutions, from 1550 × 900 and down to 30 × 20 pixels for vehicles and plates, respectively.
• Our LP detection and recognition study is beneficial for beginners and researchers who aim to conduct research in machine learning to achieve various object detection and recognition tasks for their desired applications.

The rest of this paper is organized as follows: Section 2 presents a literature review of various LP detection and recognition methods. Section 3 briefly describes the four methods that are investigated in this study. Section 4 presents detailed simulation results along with a discussion. Finally, conclusions and future works are listed in Section 5. For the ease of readers’ understanding,

Table 1. Nomenclature.

Acronym	Meaning
ALPR	Automatic License Plate Recognition
ALPDR	Auto License Plate Detection and Recognition
CNN	Convolutional Neural Networks
CRNet	Character Recognition Network
${(L}_c)$	Cross-Entropy Loss
DL	Deep Learning
DNN	Deep Neural Networks
DOE	Design Of Experiment
DPOD-NET	Deformation Planar Object Detection Network
EGSA	Edge-Guided Sparse Attention
ITS	Intelligent Transportation Systems
LPs	License Plates
LPD	License Plate Detection
LPR	License Plate Recognition
LSV-LP	Large-Scale Video-based License Plate
LSTM	Long-Short-Term-Memory
OCR	Optical Character Recognition
RNN	Recurrent Neural Network
SAC	Sparse Attention Component
ViBe	Visual Background Extractor
YOLO	You Only Look Once

lists the nomenclature that is used in this manuscript.

2. Related Work

This section reviews current techniques to detect and recognize LPs. Both LP detection and recognition are an integral part of an ITS. Normally, LP detection is followed by the recognition module. Therefore, this section describes the recent methods to achieve both LP detection and recognition tasks.

In [6], the proposed ALPR system uses CNNs, along with preprocessing and image processing morphological operations to achieve LP detection and recognition in various scenarios. It efficiently recognizes multi-line and multi-font license plates and yields good performance in the night mode by achieving a 98.13% accuracy. In [7], the authors present the VSNet-based ALPR method, which uses a resampling-based cascaded framework integrated with a plain CNN that obtains improved LP detection and recognition accuracy. The approach influences vertex information for better recognition and utilizes a weight-sharing classifier to address data constraints. The VSNet combines two CNNs, which are the VertexNet for detection and the SCR-Net for recognition using horizontal encoding and vertex-estimation techniques. In [8], a multi-style license plate recognition method is proposed by applying a feature pyramid network (FPN) with instance segmentation. This method is specifically proposed for the Hong Kong Bridge where vehicles have substantial variations in LP styles. This method simultaneously locates and classifies LP characters. The experimental results demonstrate a 98.57% recognition accuracy on multi-style license plates. To achieve robust ALPR, in [9], a novel concept of partial character reconstruction is discussed. This method is developed based on the characteristics of the stroke width in the Laplacian and gradient domain in a novel way. Meanwhile, the angular information of a character is analyzed by applying the PCA. Ultimately, this method influences the stroke width in the Laplacian and gradients to achieve final LP segmentation. In [10], the proposed ALPDR method tackles layout dependence and reduction in accuracy under unpredictable environmental conditions. This method uses lightweight, anchor-free detection networks that are encouraged by CenterNet and attention-based recognition networks with residual deformable blocks. Experiments on different datasets explain that the method outperforms conventional techniques and proves efficient in different scenarios.

In [11], the developed method discusses ALPDR of various vehicles and cars in natural scene images by using the DNN, which locates and recognizes various LPs in a single forward pass and is trained end to end. This method solves LP detection and recognition tasks by a single network and avoids intermediate error accumulation. This technique reduces error and boosts processing. Assessments on three different datasets justify the efficiency of the proposed method. In [12], the proposed ALPR method uses adjustable parameters for LP detection, character segmentation, and recognition. A major development in this method is edge clustering for accurate plate detection, the utilization of the maximally stable extreme region (MSER) detector to segment character, and an improved bilayer classifier for improved recognition.

In [13], the proposed ALPR method utilizes extremal regions (ERs) along with hybrid discriminative restricted Boltzmann machines (HDRBMs) to achieve robust LP recognition. The technique involves preliminary LP detection by utilizing top-hat transformation, vertical edge detection, and morphological operations. Initially, character-specific ERs are extracted, followed by character segmentation and fine-tuning of the LP detection method. A preprocessed HDRBM classifier is used for character recognition. The method shows durability to changes in illumination and weather conditions. In [14], the proposed model combines a fully convolutional network, which extracts hierarchical features, handles pixel-level classification, and detects multi-scale LP objects in an intelligent manner. Moreover, the AdaBoost cascade classifier is used for character segmentation. This work also addresses character recognition by utilizing an extensive learning system that is stacked with several auto-encoders for letters and digits. In [15], a two-stage ALPDR is proposed, which applies an improved YOLO-v3 tiny for LP detection. This technique also uses a lightweight MRNet for character recognition and data augmentation and RNNs that improve LP detection and recognition speed. The authors report a high LP detection and recognition accuracy, up to 99.8%, on the CCPD dataset. In [16], the proposed technique uses CNN and kernel-based ELM (KELM) to extract and recognize Chinese LP characters. This approach is also comparable to the traditional CNNs with Softmax and provides competitive results with quicker training times. In [17], the proposed method uses generative adversarial networks (GANs) for super-resolution in combination with the YOLO for LP detection and recognition. This method is extensively tested on a 60 × 60-pixel-resolution LP and achieves higher accuracy than a few other methods.

In [18], the CNN- and the MD-YOLO-based architecture is developed, which achieves real-time LP detection and handles rotated LPs. The method also addresses LP angle prediction and illustrates excellent accuracy on low-power devices. In [19], a robust edge-guided sparse attention (EGSA) mechanism is developed for real-time LP detection. The EGSA comprises two components: the edge-guided component (EGC), which enhances the LP edge, and the sparse attention component (SAC), which integrates the relevant features to portray long-range dependencies. Assessments on different datasets show that this method achieves higher precision than several compared methods. In [20], a new end-to-end algorithm is developed that uses a shared encoder to achieve robust LP detection in unconstrained traffic scenes. This work also utilizes regression networks to further refine localization. Experiments conducted on the LPST-110K dataset reveal the superiority of the developed method. In [21], the authors present a detailed comparative analysis of conventional and state-of-the-art shadow detection and removal algorithms. Moreover, a hybrid approach is developed in this work that utilizes a combination of several conventional and deep learning methods for shadow detection and removal for accurate vehicle localization. In [22], the authors construct a dataset named the LSV-LP, which has 1402 videos with 401,347 frames and 364,607 annotated LPs. The researchers have also designed a novel framework, which they refer to as the MFLPR-Net, that integrates features from adjacent and examined frames.

In [23], an ALPRNet for nonuniform LPs is proposed, which employs two fully convolutional one-stage object detectors. An assembly module is also deployed in the later stages. By focusing on bounding boxes for LPs, it avoids the RNN branches of the OCR in existing recognition techniques. In [24], the authors focus on optimizing the DOE for training parameters in transferring the YOLOv3 model specifically for license plate detection. This approach reduces DOE run requirements while providing insights into YOLOv3 parameter tuning, beyond simply enhancing training conditions. The results demonstrate that DOE improves the YOLOv3 model for LP detection. In [25], the authors propose an ALPR system for tilted and oblique LPs. This work introduces IWPOD-Net, which detects LP corners under nonuniform conditions and allows for warping to a front-parallel view to reduce perspective distortions. The results show that the proposed approach achieves top scores on several datasets.

In [26], the proposed technique uses the LPDNet and the CRNet to achieve real-time LP detection and recognition. In this work, the LPDNet utilizes an anchor-free approach to detect both the bounding box and the four corners of LPs in unconstrained environments. The Gaussian kernel and centrality loss enables the LPDNet to improve performance in complex environments, while the CRNet, which is a fully convolutional network, extracts features and combines them with the LP character classification module to achieve high accuracy in complex environments. In [27], the proposed ALPR method uses an enhanced visual background extractor (ViBe) algorithm along with fuzzy matching. Experiments conducted on the authors’ own collected dataset report 98.3% and 97.8% LP detection and recognition accuracy, respectively. In [28], a detailed review of recent trends in LP detection, segmentation, and recognition is discussed, with special emphasis on the DNN-based approaches, particularly the CNN, the RNN, and the LSTM for object detection and recognition. In [29], a novel deformation planar object (DPOD-NET) method is proposed that corrects deformed LPs by detecting corner points. This method also reduces perspective distortions by adjusting the LP visibility to a frontal view. It also proposes the LPWing loss function, which, unlike the standard L1 loss, is differentiable at zero and has a larger gradient for smaller errors. Moreover, this work introduces a stochastic multi-scale image detail-boosting strategy that enhances the dataset diversity. In [30], the proposed LP detection method uses the aspect ratio and the GrabCut algorithm. To further reduce the LP area noise, this method uses the Wiener filter combined with the Bernsen algorithm. Experiments conducted on the CCPD dataset indicate that this method yields a mean LP detection accuracy of 99.34%. In [31], an end-to-end CLPD method is proposed, which separates the LP foregrounds and backgrounds. The proposed CLPD robustly squeezes feature clusters and extracts useful discriminative features. The authors report that this method achieves much higher LP detection accuracy than several detectors on three different datasets.

It is important to note that the literature described above evaluates machine learning algorithms on different vehicles and license plate datasets.

Table 2. Summary of a few recent methods.

Ref	Technique	Year	Merits	Datasets Used	Investigations
[6]	CNN architecture	2021	Real-time operation Simple architecture	The authors’ self-collected dataset	Multi-line and multi-font LP detection Applications to night vision
[7]	CNN+Cascaded	2022	Real-time and accurate identification	AOLP CCPD CLPD PKU	Standard datasets Varying LP appearances Different day times
[8]	RPN+Faster-RCNN	2021	Slightly over 98.50% accuracy	PKU	China, Hong Kong, and Macao multi-style LPs
[10]	CenterNet and attention-based networks	2022	Simple and lightweight backbone network to extract features	AOLP CCPD VBLPD	Standard datasets Varying LP appearances
[15]	Yolov3-tiny	2023	Processes 751 FPS	PKU, MPDR, CCPD, CLPD	Scalable with the ability to process vast data
[22]	Large scale video	2023	Scalable and fast	The authors’ self-collected dataset	Video-based detection + recognition
[24]	End-to-End	2024	Contrast LP-based	ALPR, CCPD, and UFPR	Single category detection tasks

provides a summary of some recent methods along with the vehicles/LP datasets used therein. Readers are referred to the description of these papers to further investigate the nature and collection of these datasets.

The above literature depicts the importance of LP detection and recognition in various scenarios. In developing countries like Pakistan, there are several challenges before an LP detection and recognition algorithm can be used, for instance, a lack of federal standardization of LPs in the whole country. As shown in the last column in Figure 1, there are province- and capital-based LP standard templates, which are often ignored. Similarly, due to fancy LPs used as shown in the 3rd column in Figure 1, the same color of the vehicle and that of LP and its characters also poses a significant challenge to any detection and recognition algorithm. Similarly, occlusion, tilted plates, and low-resolution LP characters, as shown in the 4th column in Figure 1, are also a challenge for any algorithm. Moreover, as shown in the 2nd last column in Table 2, there have been several datasets gathered recently to address these issues. Most of these datasets are also publicly available. Our study indicates that most of these datasets have been captured by a single source in which most of the images have been acquired by a single camera. Owing to these serious challenges, this paper investigates the detection and recognition accuracy of four algorithms. Out of them, two algorithms focus on LP detection, while the other two investigate recognition. These algorithms have been investigated on the authors’ own collected dataset. To further create diversity in the dataset, the images collected have been acquired through several cameras. Below, we briefly discuss the methods that are compared in this study.

3. Methodology

In this section, we will explore the LP detection and recognition methods. Our study includes the detection and recognition performance of four algorithms. For readers’ understanding, we divide this section into LP detection and recognition segments with a brief description of each algorithm therein.

3.1. LP Detection

We investigate the performance of two LP detection algorithms on challenging Pakistani plates. Below, we briefly describe these algorithms.

Faster-RCNN-Based LP Detection [32]: This method achieves LP detection in two main phases, which are vehicle detection followed by LP localization. In this method, the Faster R-CNN algorithm is fine-tuned to extract regions of interest (RoIs) from input images to extract the region proposal network (RPN) that predicts potential vehicle regions. The process involves applying convolutional layers to generate feature maps, which are used to propose candidate regions. The feature map ψ is derived from the convolutional layers, and multiple anchors are applied to ψ to predict potential vehicle regions. The anchors are defined with various scales and aspect ratios, for example, with three scales (${128}^2$,${256}^2$,${512}^2$ pixels) and three aspect ratios (1:1, 1:2, 2:1).

To refine the proposed regions, non-maximum suppression (NMS) is used. The NMS process involves eliminating redundant proposals based on an intersection over union (IoU) threshold. For example, an IoU threshold of $\mathrm{I}\mathrm{o}\mathrm{U}\ge 0.5$ ensures that overlapping regions are reduced. Once vehicles are detected, the license plate localization module (LPLM) is developed by the authors to locate LP position. The LPLM initially transforms the RGB image to the hue saturation value (HSV) color space and uses the HSV notations as shown in Equations (1)–(3).

$$H=\left\{\begin{array}{c}60\times {\displaystyle \frac{G-B}{max-min}}+360 if max=R and G\ge B\\ 60\times {\displaystyle \frac{B-R}{max-min}}+120 if max=G\\ 60\times {\displaystyle \frac{R-G}{max-min}}+240 if max=B and G<B\end{array}\right.$$

(1)

$$S=\left\{\begin{array}{c}0, if max=0 \\ {\displaystyle \frac{max-min}{max}}, otherwise\end{array}\right.$$

(2)

$$V=max$$

(3)

After the RGB to HSV transformation, the next step is a morphological operation in which LP blobs are enhanced for this. Dilation is performed, followed by closing. Finally, the spatial area (SA) and the aspect ratio (AR) of the connected regions (CRs) are analyzed to determine if they match the expected dimensions of a license plate. The spatial area is computed as the number of pixels within a bounding box, and the aspect ratio is the height-to-width ratio. By using these techniques, the method effectively localizes the license plates within vehicle images, ensuring reliable and accurate detection. This method was developed by one of the authors of this manuscript and was previously tested on the standard PKU dataset. To learn more about this technique, please see [32] for full details.

End-to-End (E2E)-Based LP Detection [33]: This is a novel, end-to-end, and anchor-free network that simultaneously addresses the LP location and recognition. This approach introduces a lightweight ResNet-18 backbone to extract features, along with channel reduction and feature pyramid networks (FPNs). The FPN combines feature maps from various levels to create a comprehensive semantic visualization. The network architecture consists of a location branch that employs a detection head for bounding box and corner regression. The bounding box is represented by the center point ($x_c,y_c)$ and dimensions $w$ and $h,$while corners are regressed relative to this center. This setup avoids the use of the NMS by directly predicting the coordinates of the bounding box and corners. Specifically, the bounding box and corners are predicted through heatmaps, with the corners’ predictions providing precise plate localization. The RoI align layer is used to crop and resize feature maps based on bounding boxes, ensuring accurate feature extraction despite perspective distortions. The rectification process uses projective transformation to map the distorted license plate features into a rectified view, achieved by the transformation matrix as shown by (4).

$$\left[\begin{array}{c}\acute{x}\\ \begin{array}{c}\acute{y}\\ 1\end{array}\end{array}\right]=\left[\begin{array}{ccc}{a}_1& a_2& b_1\\ a_3& a_4& b_2\\ c_1& c_2& 1\end{array}\right]\left[\begin{array}{c}x\\ \begin{array}{c}y\\ 1\end{array}\end{array}\right]$$

(4)

The recognition branch employs a CNN network with 5 convolutional layers and 2 max-pooling layers and applies the connectionist temporal classification (CTC) using (5).

$$L_r=-\sum _{\left(x,z\right)\in S}\mathit{ln}P\left(z\right|x)$$

(5)

where $P\left(z\right|x)$ is the probability of the output sequence $z$given the input$x$. The decoding process uses beam search to balance search time and accuracy. For optimization, the location loss $L_d$ and recognition loss $L_r$are combined, with a weight factor $k$ set to 10. The location loss consists of several components, including focal losses for center and corner predictions, and L₁ losses for bounding box dimensions and corner offsets. The Gaussian heatmaps for labeling are computed as:

$$Y=\mathit{exp}(-{\displaystyle \frac{{(x-x_c)}^2+{(y-y_c)}^2}{2r_c^2}})$$

(6)

where $r_c$ is the standard deviation, proportional to the size of the license plate. Overall, the method presents a real-time, end-to-end solution for LP detection that operates effectively in unconstrained scenarios, combining accurate localization with efficient recognition.

We have selected this method in our comparative study because our literature review indicates that it has a simpler and more effective pipeline compared with the two-step LP detection and recognition solutions. Moreover, in this algorithm, an anchor-free method is designed to efficiently predict both bounding boxes, which is handy for unconstrained environments. Furthermore, this method is an efficient algorithm that works in real time and uses a lightweight backbone that enhances its efficiency. We also found that this method is more efficient compared with several RNN-based methods. For the implementation task, we followed the same settings as described in [33]. Readers are referred to [33] for complete details of this LP detection method.

3.2. LP Recognition

We investigate the performance of two LP recognition algorithms on challenging Pakistani plates. Below, we briefly describe these algorithms.

DNN-Based LP Recognition [34]: This method achieves LP recognition in three key distinct steps, which are vehicle detection and LP localization followed by recognition. The vehicle detection module utilizes a fine-tuned Faster R-CNN to accurately identify vehicles within an image. The detected vehicle image is processed by the LPLM to detect the LP within the vehicle’s bounding box by converting the RGB image to HSV components. Meanwhile, several thresholds are applied to each HSV component to segment the colors relevant to the LP area. Finally, morphological operations, including dilation $\left(D\right)$ and closing $\left(E\right)$, are applied using Equations (7) and (8), which enhances the LP blobs.

$$D\left(x,y\right)=\underset{(i,j)\in K}{\max }\{I(x+i,y+j)\}$$

(7)

$$C\left(x,y\right)=D\left(E(x,y\right))$$

(8)

where $K$ is the kernel and $I(x,y)$ is the input image pixel. This step smooths the contours and fills holes in the segmented license plate. The LP recognition module focuses on recognizing characters using a DNN. The LP region is extracted and preprocessed by resizing the LP image to a fixed size. The DNN then predicts the characters present on the LP features. Finally, post-processing that includes error correction is applied to refine the LP area. This method was developed by one of the authors of this manuscript and was previously tested on the standard PKU dataset and AOLP, CCPD, and the PKU datasets. Complete details of this LP recognition method can be found in [34].

CA-CenterNet-Based LP Recognition [35]: This method uses a corner-aware CenterNet (CA-CenterNet) architecture by resizing input image to 512 × 512 pixels propagated through a ResNet50 backbone. The network outputs three components: a heatmap for detecting the center of the license plate, an offset map for adjusting the center points due to downscaling, and four vectors pointing to the corners of the license plate. The heatmap, represented as $P$, is generated using a 2D Gaussian map ${(G}_m)$ using Equation (9).

$$G_m=\exp (-{\displaystyle \frac{{{(p}_x-c_x)}^2+{{(p}_y-c_y)}^2}{{\sigma }^2}} )$$

(9)

where σ is the standard deviation related to the LP size. The focal loss ${(L}_h)$ is employed to handle class imbalance, defined in (10). For offset correction, the smooth $L_0$ and $L_1$ are used as defined by (11) and (12).

$$L_h=-{\displaystyle \frac{1}{N}}\sum _i\left\{\begin{array}{c}{{(1-\widehat{p}}_i)}^{\alpha }log\left({\widehat{p}}_i\right), p_i=1\\ {(1-p_i)}^{\beta }{{(\widehat{p}}_i)}^{\alpha }log\left({\widehat{p}}_i\right), otherwise\end{array}\right.$$

(10)

$$L_o=-{\displaystyle \frac{1}{N}}\sum _{k=1}^{N}SmoothL1(o_k-{\widehat{o}}_k)$$

(11)

$$Smooth L1\left(x\right)=\left\{\begin{array}{c}{0.5x}^2, \left|x\right|<1\\ \left|x\right|-0.5, otherwise\end{array}\right.$$

(12)

In the LP character recognition module, the detected LP image is resized to 128 × 32 pixels and processed through a series of convolutional blocks. A segmentation-free model with global average pooling and eight fully connected branches predicts each character’s position on the LP. The categorical cross-entropy loss ${(L}_c)$ is applied using (13).

$$L_c=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n\sum _{j=1}^8\sum _{k=1}^m{y}_{ijk}\mathrm{l}\mathrm{o}\mathrm{g}{(\widehat{y}}_{ijk})$$

(13)

where $y_{ijk}$ and ${\widehat{y}}_{ijk}$ are the ground truth and predicted labels, respectively. The system supports multiple types of license plates with up to eight characters and is designed to work with varying LP formats, including those for vehicles using new energy. We selected this algorithm in our comparative study because of its robustness to accurately locate the multi-styled LPs and its suitability for unconstrained environments. Moreover, this is a simple segmentation-free LP recognition network that works in real time. For the implementation task, we followed the same settings as described in [35]. More details of this LP recognition method can be found at [35]. Moreover, in [36], researchers explore several deep learning techniques, such as MobileNet-V2 and YOLOx, for vehicle identification and YOLOv4-tiny, Paddle OCR, and SVTR-tiny for LP number recognition. Their developed system was rigorously tested at the Firat University’s entrance with a thousand images captured under various conditions, for instance, fog, rain, and low light. Readers’ are encouraged to the aforementioned manuscript for complete theoretical and implementation details.

4. Simulation Results

This section presents the experimental setup, a brief description of the used datasets, LP detection and recognition results, a discussion, and observations in detail. Simulations were executed in Python 3.11 with an HP Omen Intel, California, core-i7 14th generation machine with 32 GB of RAM. The dataset was divided into 80:20 for training and testing, respectively.

4.1. Dataset Description

We performed experiments on our collected Pakistani vehicle dataset, which contained a total of 16,521 images of multiple objects, such as cars, motorcycles, and pedestrians. Example images of the few images are shown in Figure 2.

The complete details of all collected objects in this dataset are shown in

Table 3. Dataset statistics.

Objects		Low Density	Resolution	High Density	Resolution
Classes	Cars	8457	88 × 57~66 × 36	655	80 × 50~66 × 36
	Vehicles		85 × 50~66 × 36		75 × 50~66 × 36
	LTVs		85 × 50~66 × 36		45 × 50~66 × 36
	Motorcycles	4136	80 × 50~60 × 40	136	60 × 50~60 × 40
	Persons	3025	50 × 80~20 × 40	112	40 × 60~20 × 40
Total		15,618		903

. In our dataset, we consider the image to be high-density whenever it contains more than two objects per image. The dataset, along with all details, will be made publicly available at (http://research.cuiatd.edu.pk/admin/login.aspx, accessed on 6 September 2024). Our collected dataset has a total of 16,521 objects that include a variety of objects, which range from cars and vehicles to LTVs and motorcycles with visible LPs. For the low-density category, there are a total of 8457 images of cars, vehicles, and LTVs whose resolution varies from 85 × 50 to 66 × 36. For motorcycles and persons, image resolution varies from 80 × 50~60 × 40 to 50 × 80~20 × 40 pixels, respectively. For the high-density category, there are a total of 655 images of cars, vehicles, and LTV whose resolution varies from 80 × 50 to 66 × 36; whereas for motorcycles and person, image resolution varies from 60 × 50~60 × 40 to 40 × 60~20 × 40 pixels, respectively. As shown in Figure 2, most of these images have been captured in open and outdoor environments with different cameras. A few of those images have also been acquired through smartphone cameras. Our collected dataset has widely varying and complex backgrounds with randomly distributed visible objects. Several of the images have been captured on main highways with a few obstacles, such as sign boards and trees. Similarly, several images have been captured from unorganized Pakistani parking lots of malls and shops. In such scenarios, vehicles are parked in a random manner and, often, in open environments. All such factors make our collected dataset entirely different from standard datasets in which a standard and a high-resolution camera captures the images of vehicles and other objects. An image labeling tool (https://github.com/heartexlabs/labelImg, accessed on 6 September 2024) was used to label and annotate the images.

4.2. Evaluation Measures and Settings

Normally, the LP detection and recognition are analyzed through false positives (fp) and miss rates. Few researchers prefer to reduce the miss rate at the cost of more false positives, which are discarded in later stages. Similarly, the LP recognition performance is also measured by the false acceptance rate and the false rejection rate. This paper reports LP detection and recognition accuracy comparison, as defined by (14).

$$Accuracy={\displaystyle \frac{tp+tn}{tp+fp+tn+fn}}$$

(14)

where $tp, tn$, and $fn$ denote the true positive/negative and false negative, respectively.

Table 4. Simulation environment settings.

Parameter	Simulation Environment
Test image	Varied from 1550 × 900 to 600 × 400 pixels
Optimizer	SGDM
Learning rate	${10}^{-4}$
Validation frequency	50
Epochs	50
Batch size	32
L2 regularization	${10}^{-4}$
Gradient threshold	L2 normalization
Shuffle	Every epoch
Momentum	0.90

lists the experimental settings for the LP detection and recognition algorithms.

In the next section, we discuss in detail the performance of implemented methods for LP detection and recognition. Moreover, a detailed discussion of our findings is also presented. For a fair comparison, we also compare the computational complexity of the four compared methods.

4.3. LP Detection Analysis

As described above, LP detection becomes hard for state-of-the-art algorithms in unconstrained environments, such as nonstandard LPs, illumination variations, and varying image resolutions. Figure 3 shows the LP detection results for the Faster-RCNN and the E2E methods. Important observations from Figure 3 are listed below in order.

• Both the Faster-RCNN and E2E methods successfully detect one LP that appears in the input image. However, as shown in the second image of the first row in Figure 3b, there is false LP detection around the backlight of the car. Both these methods successfully handle LP detection from the frontal to angle variation.
• As shown in the second row in Figure 3, both the Faster-RCNN and the E2E methods successfully detect whenever there are up to three LPs that appear in the input image. In this case, the LP shooting angle does not affect the accuracy of these methods.
• Figure 3 also indicates that the background of either the vehicles or the LPs does not affect the detection capabilities of these methods. Both these methods are robust to successfully locate the blurry LP area, as shown in the first image in the second row in Figure 3.

4.4. LP Recognition Analysis

As previously mentioned, LP recognition also presents significant challenges to state-of-the-art algorithms, particularly in unconstrained environments. Figure 4 illustrates some of the LP recognition results of both DNN- and CA-CenterNet-based methods. Key observations from Figure 4 are listed below.

• Both the DNN- and the CA-CenterNet-based methods successfully recognize a single LP that appears in the input image. Therefore, both the DNN and the CA-CenterNet methods successfully identified LPs on vehicles of different makes and orientations, highlighting the system’s versatility in real-world scenarios.
• In Figure 4a, the LPs that display characters ICRC 7219 and AGC 43 were accurately recognized in single-class LP recognition tasks by both of these methods. For multi-class LP recognition, as shown in the second row in Figure 4a, the system effectively handled nonstandard LP formats and additional text elements, such as the detection of 111 on the rear window of the white car in the second to last image in Figure 4a second row. We observed that images with embossed LP text were not recognized by either of these methods. One such example is the last image in the second row in Figure 4, which is the third car’s plate not processed by either the DNN or the CenterNet methods.
• Figure 4 also indicates that both DNN- and the CA-CenterNet-based methods are barely affected by the LP shooting angle and perform well with multi-style and different fonts.

4.5. Discussion

The above analysis sheds a detailed light on the performance of the detection and recognition algorithms. However, the discussion below will give more insights into the performance of the compared methods.

Table 5. LP detection and recognition summary.

Object Type	Method	No. of Objects	LP Resolution (Pixels)	Accuracy %	Mean Accuracy %	Overall Mean Accuracy %	Observations
LP Detection	Faster-RCNN	1	66 × 36	100	98.35	98.41	Both LP detection methods are robust. When generally tested in an outdoor environment, they yield a mean accuracy of more than 98% for up to 6 LPs that are visible in an image. In all these cases, the LP resolution varies from 66 × 36 pixels to 30 × 20 pixels. Both these LP algorithms handle varying lighting conditions along with LP angle orientation.
		2	60 × 35	100
		3	56 × 35	100
		4	55 × 30	100
		5	40 × 25	96.10
		6	30 × 20	94.00
	E2E	1	66 × 36	100	98.48
		2	60 × 35	100
		3	56 × 35	100
		4	55 × 30	98.90
		5	40 × 25	96.00
		6	30 × 20	96.00
LP Recognition	DNN	1	66 × 36	100	98.90	98.93	When tested in open environments, both the DNN and the CA-CenterNet methods yield nearly 99% accuracy. Up to 3 LPs, the LP recognition accuracy is 100%. Both these methods recognize the LP from frontal to angular orientations.
		2	60 × 35	100
		3	56 × 35	100
		4	55 × 30	97.90
		5	40 × 25	97.50
		6	30 × 20	98
	CA-CenterNet	1	66 × 36	100	98.96
		2	60 × 35	100
		3	56 × 35	100
		4	55 × 30	98.40
		5	40 × 25	97.70
		6	30 × 20	97.68

provides a summary of all detection and recognition methods that are investigated in this study. Important observations from Table 5 are listed below.

• As shown in Table 5, when the LP resolution varies from 66 × 36 to 55 × 30 pixels, 100% LP detection accuracy is obtained for up to four LPs that appear in an image. Especially, Faster-RCNN accurately handles the four LPs that appear in the input image by yielding 100% detection accuracy. For low-resolution LP images that range from 55 × 30 to 40 × 25 pixels, the DNN-based method performs better than the Faster-RCNN by yielding up to 98.90% LP detection accuracy.
• Overall, both detection methods are robust and yield a mean LP detection accuracy of 98.41%, with the E2E method performing slightly better than the Faster-RCNN by yielding a mean detection accuracy of 98.48% across all image resolutions, as shown in Table 5.
• For LP recognition and up to three LPs, both the DNN and the CA-CenterNet yield 100% accuracy for image resolution of 66 × 36 to 56 × 35 pixels. For low image resolution of up to 30 × 20 pixels, the CA-CenterNet performs slightly better than the DNN-based method.
• Overall, the LP recognition pool delivers a mean accuracy of 98.93%, with both the DNN and the CA-CenterNet methods yielding a mean accuracy of 98.90% and 98.96%, respectively.
• Generally, both LP detection and recognition pools are robust and yield a combined mean accuracy of 98.67%. These methods were tested in general and outdoor environments that varied from sunny days to rainy days, along with various image-capturing timing from dawn to early sunset. Therefore, our study concludes that all four algorithms that are used in the pools of LP detection and recognition are reliable as they yield high detection and recognition accuracy. In addition, these algorithms are also scalable as their performance does not drop when the number of LPs in the image is increased along with the decreasing image resolutions. Furthermore, all four methods used in the LP detection and recognition pool are consistent and yield over 98% accuracy for nonuniform and outdoor-captured LP images.
• Our study indicates that most of the LP detection and recognition systems are designed based on vehicle detection that has the same LP standards. There are several other methods that process multi-style LPs in different countries. Therefore, we recommend that if an LP detection and recognition system is to be developed for unconstrained environments, the E2E method can be used to locate LPs, while the CA-CenterNet-based method can be used for recognition purposes. If fine-tuned and used in a pipeline for detection and recognition purposes, these methods will yield further encouraging results. Moreover, their performance also indicates that these methods can be reliably used in many real-time applications that require quick and accurate LP detection and recognition. Moreover,

  Table 6. LP detection and recognition using high- and low-density datasets (NP denotes Not Processed).

  Methods	Low Density				High Density
  	Objects Type	Total Objects	Resolution	Acc %	Objects Type	Total Objects	Resolution	Acc %
  Faster-RCNN	Cars/Vehicles/LTVs	8457	As indicated in Table 3	98.70	Cars/Vehicles/LTVs	8457	As indicated in Table 3	98.36
  	Motorcycles	4136		98.00	Motorcycles	4136		98.35
  	Persons	3025		NP	Persons	3025		NP
  E2E	Cars/Vehicles/LTVs	8457		98.80	Cars/Vehicles/LTVs	8457		98.50
  	Motorcycles	4136		98.15	Motorcycles	4136		98.46
  	Persons	3025		NP	Persons	3025		NP
  DNN	Cars/Vehicles/LTVs	8457		99.20	Cars/Vehicles/LTVs	8457		98.90
  	Motorcycles	4136		98.60	Motorcycles	4136		98.90
  	Persons	3025		NP	Persons	3025		NP
  CA-CenterNet	Cars/Vehicles/LTVs	8457		99.30	Cars/Vehicles/LTVs	8457		99
  	Motorcycles	4136		98.70	Motorcycles	4136		98.95
  	Persons	3025		NP	Persons	3025		NP

  shows the complete breakdown of accuracies obtained for various numbers of images that are collected in our dataset. As can be seen, for each of the low- and high-density object collections, the compared algorithms perform well and all instances yield over 98% detection and recognition accuracy.

4.6. Computational Complexity

We report the computational complexity in terms of the time required to yield the output image. Figure 5 shows the times consumed by algorithms used in LP detection and recognition pools on various image resolutions. A few important observations from Figure 5a,b are highlighted below.

• Figure 5a shows the detection time for five different image resolutions for both LP detection and recognition methods. As shown in Figure 5a, the computational time for both detection and recognition methods significantly decreases as the image resolution is reduced. For instance, for the DNN method, the mean detection time decreases from 3.91 s at a resolution of 1550 × 900 to 1.78 s at 600 × 400. Similarly, the mean recognition time for the E2E method drops from 3.25 s at 1550 × 900 to 1.5 s at 600 × 400-pixel test image resolution. This indicates that lower image resolutions require substantially less processing time, which can be crucial for real-time applications where speed is essential.
• As shown in Figure 5a, the CA-CenterNet consistently shows the fastest processing times across all resolutions. For instance, it takes 2.5 s to process a 1550 × 900-pixel resolution image, while Faster-RCNN, the slowest, takes 4.8 s. Even at the lowest resolution of 600 × 400, CA-CenterNet processes the image in 1 s compared with Faster-RCNN’s 2.29 s. This makes CA-CenterNet a more efficient choice for tasks where quick detection and recognition are required, particularly at varying image resolutions. Out of the four methods shown in Figure 5a, we observed that the E2E method ranks second, while the DNN method ranks third, respectively, during LP detection and recognition tasks.
• As shown in Figure 5b, the LP detection pool takes 4 s to nearly 2 s to process image resolutions that vary from 15,550 × 900 to 600 × 400 pixels. At the same time, the LP recognition pool consumes from 3 s to slightly over 1 s for the aforementioned test image resolutions. This trend highlights the efficiency gained by working with lower-resolution images, which can significantly reduce the computational burden, particularly in scenarios where processing time is critical. The mean LP detection and recognition time varies from 3.5 s to 1.5 s.
• The execution time statistics presented above highlight the importance of selecting the appropriate algorithm and resolution based on the specific computational requirements and constraints of the task at hand. Therefore, we suggest that if real-time LP detection is desired, then E2E LP detection method is feasible. Similarly, for the task of real-time LP recognition, the CA-CenterNet-based method is recommended.

4.7. Further Analysis

As can be seen in Figure 6, all four methods have high mean performance, which is generally over 92% for all four methods. In this case, the CA-CenterNet-based method has the highest mean of 97%, followed by the E2E method, which has 96%. The mean values for the DNN and Faster-RCNN are found to be 95.83% and 91.91%, respectively. In Figure 6, we also observe the variability in all four methods in terms of the standard deviation (SD). For the Faster-RCNN and the E2E method, the SD values are found to be 58.90 and 57.87, respectively. Moreover, for the DNN- and the CA-CenterNet-based methods, the SD values are 52.77 and 50.81, respectively. A lower SD values imply that the results are thoroughly clustered around the mean and indicate that the method performs reliably under dataset variations. On the other hand, a high SD suggests greater variability in the results, implying that the method’s performance varies more across different scenarios. This inconsistency could be problematic for applications where stable and predictable outcomes are important. Since our collected dataset is gathered through multiple sources, it has substantial variations in illuminations. Figure 6 also indicates that. for the LP detection scenario, the E2E method is better, while for the LP recognition, the CA-CenterNet performs slightly better than its counterparts.

4.8. Limitations

In a Pakistani traffic environment, we found several cases where both LP detection and recognition pools algorithms failed to deliver reliable results. Figure 7 shows four categories where the algorithms used in LP detection and recognition pools failed to deliver results. Moreover, the images shown in Figure 7 are extracted from 1550 × 900-pixel input images, which have LP resolution of at least 30 × 20 pixels and are zoomed for better visualization. A few important observations from Figure 7 are listed below.

• As can be seen in Figure 7a, whenever the LP was broken, the detection algorithms failed to locate the LP area. Ultimately, when this LP was fed to the recognition pool, it could not recognize the license number. Such broken LP cases are often found on common highways in a typical Pakistani environment.
• We also observed that the algorithms employed in the LP detection and recognition pools also failed to yield satisfactory results whenever the LP area was partially or fully occluded. A severe case of LP occlusion is shown in the second image of the top row in Figure 7b.
• Surprisingly, we also found many vehicles that contained LPs with embossed characters on the LP area. A few such samples are shown in the top row in Figure 7c. In such cases, both LP detection and recognition algorithms could not yield encouraging results. The top image in Figure 7c is particularly challenging due to the visible shadow, which ultimately induces nonuniform illumination and makes it hard for any algorithm to detect and recognize the LP.
• In a Pakistani environment, we also observed many LPs that had faded letters and digits, as shown in Figure 7d. In such cases, a few digits were blurred. In some cases, the LP coating paint was eroded along with screws embedded therein. Since the resolution of these LPs was also very low, this case also posed a significant challenge for the detection and recognition algorithms. In some cases, we also observed the semi-circle typed LPs on heavy transport vehicles along with colored painted boundaries with different fonts that highlighted the vehicle license number. On a few of such tested images, both the LP detection and recognition algorithms struggled to yield accurate results. These are some of the unique cases and, unfortunately, no LP database exists to train the algorithms. Therefore, future studies could also collect such datasets and develop more robust algorithms that can handle the aforementioned issues. Furthermore, future studies could also focus to gather mixed-style LPs to train and improve these compared methods along with more exploration of machine and deep learning algorithms.

4.9. Final Remarks

The detailed analysis presented above gives a fair insight into the performance of LP detection and recognition algorithms investigated in this study. We also observe that recently developed or fine-tuned deep learning algorithms achieve acceptable accuracy on standard datasets. For instance, Faster-RCNN investigated in this study reports over 99% LP detection accuracy on the PKU dataset. Similarly, the E2E method investigates different publicly available standard datasets, such as the CCPD, the AOLP, and the PKU, and reports up to 99.3% LP detection accuracy. However, when trained and tested in unconstrained environments, both these LP detection methods yield slightly over 98% accuracy. When both the detection and recognition algorithms were tested simultaneously, the CA-CenterNet-based method ranked 1st by yielding the recognition.

The DNN-based method, in the original study, investigated three standard datasets, which were the PKU, the AOLP, and the CCPD, and reported up to 99.81% LP recognition accuracy on one of the categories in the PKU dataset. The CA-CenterNet-based LP recognition method investigated the CLPD, the CCPD, the AOLO, and the PKU datasets and reported up to 99.8% LP recognition accuracy. In contrast, when tested on open door and nonuniform LP images, the DNN- and the CA-CenterNet-based methods yielded less than 99% recognition accuracy.

The LP detection and recognition in the context of big data will involve processing vast amount of information in the form of different-style vehicles and LPs. It is also more desirable for organized traffic management and surveillance, along with law enforcement. With millions of vehicles on highways, big data technologies can be helpful in collecting, storing, and analyzing very large datasets of multi-variety LP images. Moreover, distributed and cloud frameworks can also be used to process big and complex data from several sources in large cities in real time. With the growing amount of data every day, there is a huge variety of LPs, their styles, and fonts. Therefore, a universal LP detection and recognition algorithm to tackle the aforesaid issues is yet to be developed.

During this big data era, the analysis of object detection and recognition, such as license plates, is particularly important in several real-life applications, few of which include automatic traffic monitoring and autonomous toll collections. It is also very important to track criminal pursuits and keep a record of vehicle entrances and exits at various malls and public markets. An accurate LP detection and recognition is a crucial part in intelligent transportation systems (ITS). Therefore, a reliable system that can work in all weather and natural conditions has great importance for road safety, monitoring, vehicle congestion, and reducing environmental pollution. Since LP detection is achieved in prior steps, its accurate localization is very important while developing these systems. If LPs are misclassified during the detection phase, it will lead to serious errors in the recognition stage. With the availability of huge training data these days, deep learning-based methods, such as those investigated in this study, demonstrate a high efficiency compared with the traditional methods that extract limited features manually.

5. Conclusions

In this study, we presented a comparative analysis of license plate detection and recognition algorithms in unconstrained environments. To analyze the license plate detection problem in open door environments, the performance of two algorithms was compared, which are Faster-RCNN and the end-to-end method. Detailed experiments were conducted on the authors’ gathered dataset, which contained multi-styled Pakistani license plates. The study on license plate detection concludes that Faster-RCNN yields a detection accuracy of 98.35%, while the end-to-end method delivers 98.48% correct license plate detection. Both the detection algorithms yielded a mean detection accuracy of 98.41%. Moreover, an investigation of license plate recognition reveals that the DNN-based method yields an LP recognition accuracy of 98.90%, while the CA-CenterNet-based method delivers a high accuracy of 98.96% on challenging Pakistani vehicle license plates. Both the recognition algorithms yielded a mean accuracy of 98.93%. During the task of license plate detection, the end-to-end method outperforms the Faster-RCNN by consuming the least detection time, while during the task of license plate recognition, the CA-CenterNet-based method beats the DNN-based method by identifying the plate characters in less time.

Both the license plate detection and recognition algorithms struggled to process the scenarios when the license plate was either broken or faced occlusion. Similarly, the cases where embossed or faded characters were found on the plate area also posed a significant challenge to these algorithms. Therefore, for all these cases, additional studies are required to design a feasible license plate detection and recognition algorithm.