A License Plate Recognition System with Robustness against Adverse Environmental Conditions Using Hopfield’s Neural Network

Abstract

License plates typically have unique color, size, and shape characteristics in each country. This paper presents a general method for character extraction and pattern matching in license plate recognition systems. The proposed method is based on a combination of morphological operations and edge detection techniques, along with the bounding box method for identifying and revealing license plate characters while removing unwanted artifacts such as dust and fog. The mathematical model of foggy images is presented and the sum of gradients of the image, which represents the visibility of the image, is improved. Previous works on license plate recognition have utilized non-intelligent pattern matching techniques. The proposed technique can be applied in a variety of settings, including traffic monitoring, parking management, and law enforcement, among others. The applied algorithm, unlike SOTA-based methods, does not need a huge set of training data and is implemented only by applying standard templates. The main advantages of the proposed algorithm are the lack of a need for a training set, the high speed of the training process, the ability to respond to different standards, the high response speed, and higher accuracy compared to similar tasks.

1. Introduction

Nowadays, the use of surveillance-based security systems has become increasingly important in various applications, such as home security and traffic monitoring. Object detection is one of the fundamental building blocks of automated surveillance systems. Among the most used techniques for object recognition in surveillance systems is the recognition of vehicle license plates [1]. Automatic license plate recognition is an image-processing-based method that is used for security applications such as controlling access to restricted areas and tracking vehicles.

In real-world applications, simple License Plate Recognition (LPR) systems have low detection accuracy [2]. On the one hand, the effects of external factors such as sunlight and car headlights, license plates with inappropriate designs, the wide variety of license plates and, on the other hand, the limited quality of the software and hardware related to the camera, have reduced the accuracy of these systems. However, recent advances in software and hardware have made LPR systems much safer and more widespread [3,4]. A countless number of these systems are working around the world and are growing exponentially and can do more tasks automatically in different market segments. Even if the recognition is not 100%, a results-dependent side program can compensate for the errors and provide an almost flawless system. For example, to calculate the car’s parking time, from entering to leaving the parking lot, this side program can ignore some ignorable errors in the two recognitions. This intelligent integration can overcome the shortcomings of LPR and produce reliable and fully automated systems [5,6,7].

Figure 1 shows a typical configuration of an LPR system. The license plate reader software is a Windows background program on a PC and an interface between a set of cameras. The program receives the images of the cameras, and by processing them it extracts the license plates of the cars in traffic. The program then displays the results, and can also send them to other parts of the system such as a camera or LED display via serial communication. It then sends this information to the local database or external databases (through the network).

The first step in recognizing a car license plate is to distinguish the car from other objects in the image. For this, the methods presented in previous works can be used [8,9,10,11,12]. In similar works, the use of a convolutional neural network (CNN) replaces parts of the proposed method in this paper. Despite the ease of use of this new neural network, there are major disadvantages associated with CNNs. The main disadvantage is that CNNs take a much longer time to train. Another important disadvantage is the need for larger training datasets (i.e., hundreds or thousands of images), and for their proper annotation, which is a delicate procedure that must be performed by domain experts. Other disadvantages include problems that might occur when using pretrained models on similar and smaller datasets (i.e., of a few hundreds of images, or smaller), optimization issues due to model complexities, and hardware restrictions [13]. However, the proposed algorithm assumes there are only cars on highways or an absence of objects corresponding to license plates in non-car elements (such as humans, etc.). This contribution can overcome the burdens that were present in previous works.

In the literature, the pixel-by-pixel comparison method has been used to match the segments extracted from the image with the defined standard characters. This method, in addition to the low classification accuracy, lack of identification, and removal of noise, also requires a lot of execution time. We employed the Hopfield neural network [14] to simultaneously speed up the program’s execution and increase the precision, while removing noises on the segments extracted from the image. License plate recognition systems are usually used outdoors. The presence of air pollution, fog, and other factors causes the car license plate images to become blurred. By using frequency domain techniques, it is possible to remove the side- and destructive effects of the environment on the image. In this paper, first, the location of the license plate is identified, and then the disturbing effects of the environment are removed. After extracting the license plate image’s segments, the Hopfield neural network classifies these segments to corresponding defined characters.

2. Methodology and Simulation Results

Figure 2 shows the five main steps of license plate recognition. In this structure, the steps for image scheduling, camera settings, and the saving and transferring of results have been ignored. In the following, each of these five steps will be explained in full detail.

2.1. Preprocessing of the Image

In tasks based on image processing, such as car license plate recognition [15,16,17,18] or eye tracking, etc., the first step is usually to determine the approximate location of the target object. For this, a large number of different cars with different license plate locations were studied. All the studied cars were photographed at the same distance and angle. In all these photos, the place of installation of the license plates was marked. Figure 3 shows the border of the area where it was possible for a license plate to exist, taking a suitable tolerance.

The area outside the border of the license plate’s zone (which has non-useful information) will be affected by the Blur filter. Applying this filter reduces the calculations and the possibility of errors in future processing. Figure 4 shows a typical image affected by this filter, where non-useful areas have been blurred.

2.2. Elimination of Adverse Environmental Effects

Before determining the exact location of the license plate, and then its characters’ segments, the adverse effects of the environment, such as the effects of possible fog or smoke in the space between the camera and the license plate, should be corrected as much as possible. Equation (1) shows a blurry image relation [19,20,21,22,23,24]:

$$I(x)=J(x)t(x)+A(1-t(x))$$

(1)

where $I$ is the intensity of the light in the image, $J$ is the illumination of the scene, $A$ is the general light of the environment, and t is a parameter that describes the part of the light that was not scattered and reached the camera. The elimination of adverse environmental effects means recovering $J$, $A$, and t from I. The term $J(x)t(x)$ in this equation is called direct attenuation, which describes the brightness of the scene and its decay in the environment. The term $A(1-t(x))$ is called ambient light, which comes from the previously scattered light and leads to a change in the color of the environment. When the space is homogeneous, the transfer coefficient t is described as follows:

$$t(x)=e^{-\beta d(x)}$$

(2)

where $\beta$ is the dispersion coefficient. Equation (2) clearly shows that the image brightness decreases exponentially with its depth $d$.

Equation (1) shows that in the RGB color space, vectors A, I(x), and J(x) are coplanar while their endpoints are located on a single line. The transfer coefficient t can be expressed by

$$t(x)=\frac{\left|\right|A-I(x)\left|\right|}{\left|\right|A-J(x)\left|\right|}=\frac{A^c-I^c(x)}{A^c-J^c(x)}$$

(3)

where $c$ represents the index of the color channel. In blurred images, t is less than one. Thus, the resolution of the image, which is the sum of the image gradients, is low. The following illustrates this reduction:

$${\displaystyle \sum _t\left|\right|\nabla I(x)\left|\right|}=t{\displaystyle \sum _t\left|\right|\nabla J(x)\left|\right|}<{\displaystyle \sum _t\left|\right|\nabla J(x)\left|\right|}$$

(4)

The transmission coefficient, t, is estimated by maximizing the image resolution, while the intensity $J(x)$ is less than the intensity $A$. The dark channel for a haze-free outer space image is defined as the following: in a non-sky image, at least one color channel has very low brightness in some pixels. In other words, the image brightness in these pixels is minimum. Equation (5) shows the dark channel definition:

$$J^{dark}(x)=\underset{c\in \{r.g.b\}}{\min }(\underset{y\in \Omega (x)}{\min }(J^c(y)))$$

(5)

where $J^c$ is a color channel of $J$ and $\Omega (x)$ is a piece of the image centered at $x$. If the image does not include the sky and does not have fog, the intensity of $J^{dark}$ will be almost zero. Assuming the value of $A$ for the ambient light and the constant transmission coefficient $t(x)$ in a piece of the image, minimizing the intensity (1) gives:

$$\underset{y\in \Omega (x)}{\min }(I^c(x))=t(x)\underset{y\in \Omega (x)}{\min }(J^c(y))+A^c(1-t(x))$$

(6)

Dividing the sides of Equation (6) by $A$ and minimizing again, this time among the color channels gives:

$$\underset{c}{\min }(\underset{y\in \Omega (x)}{\min }(\frac{I^c(x)}{A^c}))=t(x)\underset{c}{\min }(\underset{y\in \Omega (x)}{\min }(\frac{J^c(y)}{A^c}))+(1-t(x))$$

(7)

By approximating Equation (7) to zero, the transfer coefficient can be defined as Equation (8). The coefficient w is defined to adjust the blurring in the image.

$$t(x)=1-\underset{c}{w \min }(\underset{y\in \Omega (x)}{\min }(\frac{I^c(x)}{A^c}))$$

(8)

By limiting the transmission coefficient on the limit of t₀, the brightness of the image is expressed as:

$$J(x)=\frac{I(x)-A}{\max (t(x),t_0)}+A$$

(9)

According to the above-stated contents and the presented equations, using the algorithm shown in Figure 5 it is possible to reduce the image blurring to an acceptable level.

Figure 6 demonstrates the results of the algorithm in modifying an image that was artificially and exaggeratedly fogged.

2.3. Determining the Exact Location of the License Plate

After removing the adverse environmental effects, according to the following two principles, the exact location of the license plate must be determined: First, due to the difference in the colors around the license plate and its background, by using edge detection on the black–white image the edge of the license plate frame will appear as an edge and a closed path shape. Secondly, according to each country’s standards, the length-to-width ratio of the license plate will be a fixed value.

According to the above content, all the edges on the image are detected. Detected edges become bolder to remove any interruptions in the closed paths. Then, by defining the closed path edges in the image as objects, the one with the standard license plate’s length-to-width ratio is selected as the main object (the license plate frame). Detecting the main object’s position from the initial image determines the license plate frame. Figure 7 shows all the above steps on a sample image.

2.4. Determining the Segments inside the Plate

After cutting the image of the license plate from the original image, by applying rotation if needed, removing the unessential edges of the license plate, and turning it into the black and white mode, the segments of recognizable license plate characters are separated [25,26,27,28,29].

In the image shown in Figure 8, from the left side, the index of the first column has at least one white pixel, labeled as the start index of the first segment. Additionally, the index of the first column without a white pixel is labeled the final index of the first segment. This process is repeated for the whole of the plate to determine all its character segments. Figure 9 shows the cut segments separately.

2.5. Recognizing the Segments Using Hopfield’s Neural Network

Determined character segments in the previous section should be recognized using standard character patterns. Diverse methods of pattern recognition include matching pixel by pixel [26], the k-nearest neighbor, and Bayesian, and various neural networks can be used [30]. The Hopfield neural network is known as the most common method for detecting patterns with binary features. Since the extracted black and white segment images have binary values (zero for black and one for white), they can be recognized using this neural network [31].

The main idea of the Hopfield neural network is based on state variables. If the new position of a system depends on its previous one, it can be written in terms of state variables in the form of the following equation [32,33,34,35]:

$$x(t+1)=f(x(t))$$

(10)

The sequence above continues until its energy is exhausted, and then remains in a balanced state. The system energy should decrease to reach this state. For this purpose, as shown in Figure 10, the Hopfield neural network is designed such that, firstly, the new position of the system is dependent on its previous one, and second, its energy equation decreases.

The energy function and its gradient for the system shown in Figure 10 are defined as:

$$\begin{array}{ll}E& =-\frac{1}{2}{\displaystyle \sum _i{\displaystyle \sum _j{\omega }_{ij}{x}_i{x}_j-{\displaystyle \sum _i{b}_i{x}_i}}}\\ \frac{dE}{dt}& ={\displaystyle \sum _i\frac{dE}{d{x}_i}\frac{d{x}_i}{dt}}\\ & ={\displaystyle \sum _i\frac{dE}{d{x}_i}\frac{d{x}_i}{dne{t}_i}}\frac{dne{t}_i}{dt}\\ & ={\displaystyle \sum _i\left(-\frac{dne{t}_i}{dt}\right)\frac{d{x}_i}{dne{t}_i}\frac{dne{t}_i}{dt}}\\ & =-{\displaystyle \sum _i{\left(\frac{dne{t}_i}{dt}\right)}^2\frac{d{x}_i}{dne{t}_i}}\\ & =-{\displaystyle \sum _i{\left(\frac{dne{t}_i}{dt}\right)}^2{\phi }^{\prime }\left(ne{t}_i\right)}\end{array}$$

(11)

where $\phi$ is the function of the neurons in the network and $ne{t}_i$ is the output of the ith neuron. According to Equation (11), the energy gradient of the system decreases if the derivative of the neuron function is positive. Choosing sign function as a function of the neuron can meet this condition. When Hopfield’s neural network is used as an image classifier, the two-dimensional images should be mapped to one-dimensional mode, and the black pixel values set to −1 (while the value of the white pixels are +1).

The segments which are in black and white format are resized to a standard size. The matrix of each segment, which is two-dimensional, is transformed into a one-column vector. The black pixels marked with 0 in this matrix are changed to −1 and then applied to the Hopfield neural network. On the other hand, in this neural network, the main characters in standard size and 0 values corresponding to black pixels, which are replaced by −1, are defined as balanced points. The Hopfield neural network moves the input matrix to the nearest balanced state. In other words, the closest standard character similar to the target segment is recognized. Figure 11 shows the noise (caused by mud) on the license plate. The designed Hopfield’s neural network classified this noised license plate’s character segments without error.

To determine the accuracy of Hopfield’s neural network in determining numbers and letters, according to Figure 12, a set of different car license plates was considered. The graphics on these plates can play the role of noise. Hopfield’s neural network classified the 253 characters on the license plates of this collection (after image processing and segmentation). Among the 253 test characters, only 6 characters were recognized wrongly, showing an accuracy of 97.6%. Using a computer with Core(TM) i7-2640 CPU and 8 GB RAM, the time spent on determining the characters of each license plate is about 0.08 s.

In addition to the 253 main characters (numbers or letters), there are also 26 special characters (such as a dash or a combination of numbers and letters). According to the different standards in these plates, all special characters were considered as a unit pattern. In addition to correctly recognizing the main characters, Hopfield’s neural network could also classify special characters in this unit pattern. The updated classification rate considering the special characters was 97.1%.

The agile training capability of the Hopfield neural network has made it appropriate for application to plates with different standards, while for other neural networks, such as convolutional neural networks, a huge set of training data must be collected for each standard of the plates. Moreover, the accuracy of the proposed algorithm was higher than a number of similar ones developed on SOTA [15,16]. On the other hand, the time spent on recognizing the characters of each license plate was almost equal to the time spent on recognizing only one character in methods based on convolutional neural networks [15,16,17,18].

3. Conclusions

A new license plate pattern recognition system has been presented that is robust against adverse environmental effects such as fog or mud. Unlike previous studies that only considered a certain standard of license plates, this work evaluated all objects irrespective of their types. However, the selection of objects depends on their positions, and if the recognition of the license plate characters is unsuccessful another object enters the recognition process. Additionally, the paper has addressed the challenges posed by the presence of fog and smoke in the image by removing the matter of the image before initiating the license plate recognition process. More importantly, the use of Hopfield’s neural network for license plate recognition, instead of the conventional method of pixel-to-pixel comparison of image segments with standard characters, has significantly reduced the execution time and increased the accuracy. The results pinpoint the efficacy of this approach. The neural network has shown capability in removing the noise on the license plate, making it a reliable tool for license plate recognition. The findings of this study contribute to the field of automated surveillance systems by providing an effective and efficient method for license plate recognition.