Thermal Feature Detection of Vehicle Categories in the Urban Area

Abstract

The main goal of this paper is to present new possibilities for the detection and recognition of different categories of electric and conventional (equipped with combustion engines) vehicles using a thermal video camera. The paper presents a draft of a possible detection and classification system of vehicle propulsion systems working with thermal analyses. The differences in thermal features of different vehicle categories were found out and statistically proved. The thermal images were obtained using an infrared thermography camera. They were utilized to design a database of vehicle class images of passenger vehicles (PVs), vans, and buses. The results confirmed the hypothesis that infrared thermography might be used for categorizing the vehicle type according to the thermal features of vehicle exteriors and machine learning methods for vehicle type recognition.

1. Introduction

The number of automobiles has been increasing in urban areas in the whole world for decades. It results in serious problems such as congestions, traffic accidents, and air pollution, affecting cities on a social, economic, and environmental level [1]. Electromobility offers a solution to reduce carbon dioxide emissions and become less dependent on gasoline or diesel fuels. It becomes popular in current society due to reduced energy consumption [2]. In traffic planning, management, and transport, data collection is a necessity for the continuity and safety of all vehicle types [3]. In the past, collecting traffic data was limited to manual methods or induction loop collection in stationary positions [4]. These methods neither provide spatial coverage and complex information nor capture all road user types [5]. Hence, video detection systems were developed. They are the most promising from nonintrusive methods [6]. Traffic control needs to be adapted in real time to monitor traffic flow, traffic classification, and other traffic engineering parameters in this way [7].

Vehicle types can be categorized according to their emissions standards, type of engine unit used for tolling ways, emission zones, emergency, safety systems, etc. Their categorization has to be integrated with intelligent transportation system (ITS) applications to keep them competitive in the future.

Nowadays, traffic control systems rely on conventional methods using visual cameras. Their accuracy under bad weather conditions, such as fog, snow, or heavy rain is low [5]. This is the reason to replace or complement them with other equipment, e.g., infrared cameras. Thermal images of vehicles promise to provide continuous and reliable detection of vehicles regardless of surrounding conditions [5].

Several approaches to automated vehicle detection and classification have been carried out in recent years. For example, the article [8] proposes a classification based on the length of the vehicle. A similar approach is chosen by another author [9]. Image-based methods are developed to speed up the video detection processing time. Systems for image classification based on artificial intelligence have been adopted [10,11,12,13]. Reliable detection of battery electric vehicles (BEVs) can be considered for preferences at light-controlled intersections, activation in parking and smog situations, or application of toll systems. BEV detection can support the further development of electromobility in cities.

Application of Vehicle Detection and Classification

More areas of science, industry, and common life take advantage of the power of computing in computer science. Automatic techniques for processing and classifying vehicle bodies are attractive due to their wide applicability in various real-world applications [10]. Classification and its reliable processing methods need to be adopted in real conditions, e.g., rain, fog, low picture resolution at night, or shadow contrast during the day.

Video vehicle detection systems have become popular in many agglomerations [13], and they offer an interesting alternative to other sensors [14]. Detection methods and visual object tracking have been developed in recent years [15,16,17]. Their disadvantage is the use of conventional video cameras resulting in incapability to recognize all road users. Thermal video cameras expand their potential [5]. The video system fulfills the requirements if it is able to work in varied conditions, e.g., illumination, dark, shade, or other weather conditions [16].

Therefore, commonly used methods to detect vehicle headlamps or taillights are used with day and night detection algorithms [17,18]. The Japanese team [18] mentions that it is essential to control traffic flow to detect vehicles 24/7 with high accuracy regardless of changing conditions.

Other sources refer to a comparison of video technologies, particularly conventional visual cameras and thermal cameras [5], that processed captured video using the object tracker available video in the open source project of computer vision “Traffic Intelligence’’ [19]. Individual pixels are first detected and tracked from frame to frame and saved as body trajectories using Kanade–Lucas–Tomasi tracking algorithm [20].

The article [21] presents the possibilities of the classification of vehicles based on their temperature signs. The topic focuses on the different thermal features of various vehicle categories (passenger car, van, and truck) and distinguishes them by statistical analysis of their thermal images. The detection area is an entire image with a predefined height and width. Only a part of the temperature histogram is statistically evaluated as a representative part for the vehicle category. Author [18] uses the windshield and its surroundings as a detection area. However, this method proved to be less accurate in winter due to windshield temperatures similar to ambient temperatures [22]. Authors [23] use both visual and thermal video for detection. They focus the detection area on the grille area and headlight.

Artificial intelligence (AI) and machine learning offer other promising solutions for processing image data. Authors [11] propose a methodology that uses machine learning for the classification of vehicles from a visual video at night. The authors [24] provide a solution using histograms of oriented gradients and support vector machines, which is one of the machine learning methods. Deep convolutional neural networks are currently auspicious processing techniques [24,25]. Convolution neural networks (CNNs) are deep learning algorithms that have already obtained countless results on computer vision tasks [25,26,27,28]. Pixel classification is a topic for research in computer vision. It provides different solutions such as semi-supervised learning [25] or fully convolutional architecture [29,30].

Every car generates heat while driving [21]. Even an electric engine car generates heat, although it is significantly less heat, compared to conventional vehicles. The heat is emitted to the body of the vehicle, and it is easily visible on the front mask and vehicle side, perhaps on defined places of the rear end. The goal of this paper is to present a method for reliable detection of thermal features of different vehicle categories, particularly battery electric vehicles (BEVs) and internal combustion engine vehicles (ICEVs).

2. Materials and Methods

Methods used for categorizing electric and combustion engine vehicles apply the principles of thermal analyses. The thermal and visual RGB camera was used to record vehicles passing by. The thermographic software FLIR Tools + (version 5.1, FLIR Systems, Wilsonville, OR, USA) [31] analyzed a radiometric video stream from a thermal camera. The same software separated the thermal values of pixels from the image. The thermal video stream was recorded by the FLIR E5 camera (FLIR Systems, Wilsonville, Oregon, USA). The used camera can record radiometric images in *.jpg format as well as a radiometric video stream in *.seq format, with possible conversion to *.avi format. FLIR E5 was equipped with an uncooled microbolometer detector, with a resolution of 90 × 120 pixels and speed rate of 30 fps [32], where individual pixels corresponded to the surface temperature of the measured object at a given point. The image resolution (thermogram) was determined by thermal detector resolution. The continual traffic stream was recorded by RGB camera VISION (Bucheon, Republic of Korea) with a resolution of 1280 × 720 pixels and speed rate of 30 fps [33]. The resolution of the RGB camera Hikvision that recorded traffic plates of vehicles was 1920 × 1080 pixels with a 50 fps speed rate [34].

The results of our previous measurements were presented in the previous paper [35] and they showed a significant difference between the temperature patterns of electric and combustion engine vehicles. The biggest difference in the detected released temperatures was up to dozens of degrees Celsius in a temperature-controlled laboratory environment. The tables below thermogram pictures (Figure 1) show that the highest temperature reached in the detection window and the average temperature of the detection area.

The results of electric vehicle (Figure 1) are the following:

(a)

Mild temperature affects the vehicle body in detection area Bx1, and the warmest heat source is the radiator and its grille;

(b)

Second experimentally measured area Bx2 is independently heated due to passive radar and would distort the measurement.

Figure 2 of the combustion vehicle thermogram shows a considerably affected area of the vehicle mask, primarily the grille area and vehicle hood. Temperatures are up to 40 °C higher in comparison with an electric vehicle. Respectively, the average temperature of the detection area is more than 10 °C higher by the same vehicular speed, time of drive, and the same settings of measured parameters of the thermal camera. This temperature gap confirms the results of measurements carried out on the vehicle side. The electric vehicle has heated tires and the surrounding area of the wheel arch and its fender are slightly warmed, compared to the combustion engine vehicle (Figure 2). In addition to warm tires, the wheel arch and front fender area are more affected by higher temperatures.

According to a comparison of the average and maximum temperatures of the individual thermograms, there are clear differences in the highest measured temperature in specific sections of vehicle bodies.

Different color pallets represent the temperature values (Figure 3). We used the original grayscale due to the reason of creating a thermal image database to train the neural network to classify vehicle categories.

Proposed Method of Data Processing

Categorizing vehicles into BEV and ICEV categories was achieved by setting parameters’ region of interest (ROI) on the vehicle mask. ROI differs from other parts of the vehicle by the generated engine thermal energy and other operating parts that emit energy into this area. These conclusions were confirmed in our previous work [35]. Knowing each pixel’s temperature value in the radiometric image type of vehicle engine unit was determined. Figure 4 presents the flowchart of the proposed method.

First, radiometric images of passing vehicles were manually separated from the thermal video stream. The adjustment of the temperature range of the thermal image was subsequently placed to eliminate possible errors of automatic optimization of the temperature scale. The vehicles belonged to the following categories: passenger ICEV, VAN, passenger BEV, and BUS. The correct category was validated by a visual RGB camera. Two categories, VAN and BUS, were included in the research due to their standard share in urban traffic flow. Statistical differences were detected between the thermal properties of the most represented categories of vehicles in urban traffic.

The analyses of the extracted data were subsequently placed in the file. Each detected vehicle was double-checked by manual analysis. It consisted of the selection of the ROI (i.e., vehicle mask) and extraction of the pixel values of this ROI by the thermal analysis program. The size of the ROI varies due to the size of the vehicle and its body in the radiometric image. The number of extracted pixels is directly proportional to the size of the detection window ROI. The next step was optimizing the temperature matrix for other calculations. The program for calculation and proper analysis of extracted data was written in visual basic language.

Analysis of variance (ANOVA) [37] was used for statistical evaluation. The null and alternative hypotheses were determined. No significant difference between the vehicle categories was statistically found in the case of null hypothesis confirmation. On the other hand, the alternative hypothesis confirms there is a significant difference in thermal features between the vehicle categories. The evaluation took place in statistical software StatSoft Statistica [38], which offers advanced statistical functions including ANOVA. The data were uploaded to this software. The data contain the calculated average temperature values of ROI for all measured vehicles with assigned categories. The test criterium was calculated with a prespecified threshold (0.05) of probability “p”. The post hoc Scheffe test [39] was used for a detailed confirmation of ANOVA results and furthermore to find the results among vehicle categories.

The following calculations determine the average temperatures from the extracted values of pixels and standard deviations for each detected vehicle according to their categories. Automatic categorizing of individual vehicles was based on this information. The algorithm proceeded according to the formula (1) to decide whether average temperatures (extracted and calculated the value of pixels) of ICEV or BEV are inside of the boundaries of intervals for these categories.

$$\mathrm{T}=\left[\begin{array}{ccc}\begin{array}{c}\begin{array}{cc}{t}_{11}& t_{12}\end{array}\\ \begin{array}{cc}{t}_{21}& t_{22}\end{array}\end{array}& \begin{array}{c}\cdots \\ \dots \end{array}& \begin{array}{c}{t}_{1c}\\ t_{2c}\end{array}\\ \begin{array}{cc}⋮& ⋮\end{array}& \ddots & ⋮\\ \begin{array}{cc}{t}_{r1}& t_{r2}\end{array}& \cdots & t_{rc}\end{array}\right]={ \mathsf{\mu }}_{(t_{11},t_{12},\dots .t_{rc})}$$

(1)

where t₁₁–t_rc (°C) are temperatures of each pixel in the detected window.

According to the calculated average of temperatures by this formula and standard deviation, decision-making process was set as follows:

• When calculating “σ”, if the standard deviation < threshold –”σ”, the vehicle is supposed to be BEV;
• When calculating “σ”, if the standard deviation is between thresholds –”σ“ and +”σ”, the vehicle is supposed to be ICEV.

If the calculated ROI average temperature or its standard deviation is within the temperature range of the ICEV and outside the range of the BEV, the algorithm enters logic “1” in the register. It indicates the internal combustion engine vehicle. If the temperature is within the range of the electric vehicle category, the temperature detector writes a logic “0”, which indicates the battery electric vehicle.

3. Methodology of Measurement

Image data were captured during August and December to verify the functionality of the engine thermal signature recognition. Measurements that were carried out under winter conditions proved that the reliability of detection is higher in winter. Nevertheless, difficulties might occur in the system, especially at higher outdoor temperatures from May to September in central Europe. The smallest difference between the air temperature and the temperature of the vehicle mask warmed by the engine occurs during the summer months and is therefore important to prove the validity of the evaluation. The video streams (RGB and thermal) consist of hours of recording time. Representative four hours of the video stream recorded during various light and weather conditions were selected for statistical evaluation.

Table 1. The recorded data of passing vehicles.

Category of Vehicles	Number of Vehicles	Average Temp. (°C)	Standard Dev.
Passen. ICEV	200	25.36486	2.286562
VAN	200	26.71089	2.383138
Passen. BEV	9	25.98951	2.244830
BUS	32	26.45461	3.147705
All Groups	441	26.06713	2.479222

shows representative data containing the number of passing vehicles during the four hours of measurement, the average temperatures of ROI, and the calculated standard deviation. The average size of the detection window was, for ICEV/BEV categories 25 × 15 pixels; van 30 × 15 pixels; bus 40 × 20 pixels. Multiplying the number of pixels of the simple detection window represents the number of thermal data in this window. Note that the number of recorded BEVs is not higher than two percent of the total number of recorded vehicles. This small number corresponds with the actual number of new registered BEVs in the Czech Republic [40]. The percentage in comparison with total registered vehicles in the Czech Republic (according to the official central registry of road vehicles) is small [41].

The location of the measurement was chosen near the junction exit, a street with three lanes and high traffic flow. The cameras were installed under a bridge in Argentinska Street in Prague). It is a busy road with a 24 h density of more than 31,000 cars [42]. The decision of where to position a thermal camera was based on two facts. First, a suitable position to record approaching vehicles, thus recording vehicle height and angle. Second, the video recording was less affected by weather conditions (cameras were placed under the bridge to prevent their damage in case of bad weather). The record was made in two particular positions. The cameras captured the mask and the side of the vehicle in one moment.

This measurement considered the results of previous laboratory experiments. They confirmed the hypothesis that the detection of BEV is based on the thermal signature. These results were obtained under the controlled laboratory environment without changes in lighting conditions. The measured vehicles were similar in body shapes, colors, and horsepower. Similar conditions are not possible to reach in real traffic situations. The vehicles are different in body shapes, colors, and horsepower. This affects the parameters of measurement. For example, different colors have different emissivity, and different horsepower in engines cause them to emit various amounts of heat into the vehicle mask. This might be below the resolution of the thermal camera used. Despite these limitations, thermal detection must be considered. Hence, it was necessary to confirm or decline the hypothesis made in [35] in real traffic and weather conditions. Three cameras were used for the experiment—two visual cameras, one for recording and collection of vehicle registration plates and the other for continuous recording of traffic, and a third that was a thermal camera. Visual cameras with a continuous video stream are used mainly due to the manual control of classified vehicles from the thermal video stream. The infrared thermal camera has a smaller resolution and that is why the visual camera was used to record the registration plate numbers of passing vehicles. Numbers that start with the letters “EL” are used for the recognition of electric vehicles [43] (registration plate law in the Czech Republic). That is why the visual traffic flow recording, manual classification, and separation from the thermal traffic flow were carried out (and eventually hybrid and hydrogen cars were excluded).

Procedure of Experiment with the FLIR E5 Camera and Visual RGB Camera

The following paragraphs provide a brief description of the experimental procedure to determine the optimal settings of the measuring parameters of thermal and visual RGB cameras. Thermal camera settings involve length measurement points, emissivity, atmospheric and reflected temperature, and relative humidity. Temperature range calibration is an automatic function of the thermal camera. This function is optimized, e.g., by changes in lighting conditions. The following thorough setup of parameters was performed before measurement:

• atmospheric temperature, 19 °C;
• emissivity, 0.95;
• distance, 7 m;
• reflected temperature, 20 °C;
• relative humidity, 60%.

Automatic calibration of the temperature range was not suitable for all measuring conditions. Manual calibration was not an optimal solution due to many objects (vehicles) of different emissivity in the measured area. FLIR Tools + was used to optimize the thermographic analysis.

Lens focusing was adjusted in visual cameras as well as defining areas (lanes) for vehicle license plate detection and registration. Record of the video stream of a thermal camera was saved on a laptop. Visual camera records were saved as a video stream on the internal camera memory.

Electric vehicles in the traffic flow are not common yet. A Tesla car was hired to provide examples of the drive of BEV in real operating conditions. It was preheated to operate with temperature matching after a half-hour drive in city traffic. Subsequently, Tesla was driven to pass the prepared cameras several times to obtain its images. Figure 5 shows the measurement layout.

4. Results

The normal distribution of average temperature is a condition of statistical evaluation. Histogram (Figure 6) shows the data recorded by the thermal camera and proceeded by the thermal analysis program. The analysis used the normal distribution and created a statistical test known as the analysis of variance (ANOVA). ANOVA test compared the average of four vehicle categories, specifically, passenger ICEV, passenger BEV, VAN with a combustion engine, and BUS with a combustion engine.

The statistical test was based on the confirmation of the alternative hypothesis of whether there is a significant difference between the average temperatures of the ROI mask of the vehicle for each category. The probability ”p” was lower than the prespecified threshold (0.05); therefore, it was possible to confirm this hypothesis. There was a statistically significant difference between groups, as determined by one-way ANOVA (F (3.437) = 10781, p = 0.000001).

There was a gap among these groups. Statistic post hoc Scheffe test proved that the significant difference was just between the two groups—passenger ICEV and VAN—according to the red highlighted results (

Table 2. Scheffe test results.

Category of Vehicles	Scheffe Test; Level of Significant p < 0.05000
	{1}	{2}	{3}	{4}
Passen. ICEV Average Temperature {1}		0.000001	0.900207	0.129651
VAN Average Temperature {2}	0.000001		0.854726	0.957246
Passen. BEV Average Temperature {3}	0.900207	0.854726		0.966665
BUS Average Temperature {4}	0.129651	0.957246	0.966665

).

Scheffe test proved that there was no statistically significant difference between the two remaining categories, i.e., passenger BEV and BUS; the statistically significant difference did not occur in other categories. This might be caused by an insufficient number of images of personal BEVs and buses. In total, 9 personal BEV and 32 BUS were recorded; therefore, the data do not provide sufficient evidence for comparison so far. The probability plot of average temperature (Figure 7) makes the difference well visible.

A larger set of image data containing BUS and passenger BEVs will contribute to more precise results. It is essential to continue developing this method and its accuracy. Thermal analysis is suitable for a larger proportion of BEV in the traffic flow.

5. Discussion

The results of the completed experiments showed the limitation of the camera and insufficient resolution of the thermal detector. Another issue was the low number of BEV vehicles in the traffic flow. To obtain more precise results, this method can be improved based on the hypothesis from the previous laboratory measurements [35]. The method used has its drawbacks. Various designs of vehicle engines bring uncertainty about the place of emitted thermal energy to the vehicle mask. In addition, inaccurate detection of vehicles might be caused by overlapping vehicles. To conclude, the following five key points are highlighted:

• The hypothesis confirmed in previous work in laboratory conditions was evaluated in real traffic conditions;
• The hypothesis is valid for an appropriate number of BEV vehicles in traffic flow for qualitative confirmation of all parts of the hypothesis;
• This paper provides a basic approach for the thermal detection of BEV in traffic flow;
• Varied lighting and weather conditions require a program for an automatic recalibration of thermal camera parameters;
• The convolution neural network software, vector machine, and other object detection and classification methods need to be adopted to increase the precision of the presented approach in the future.

Vehicle detection and categorization system in traffic flow might be designed based on thermal analysis and correct targeting of the thermal camera. Positive results can be achieved even in summer when the ambient air temperature approaches the temperature of the car’s heated mask. The results provide the following anticipation: if there is a difference between the categories of personal ICEV and VAN, this difference might occur in other categories. The low number of electric vehicles in the current traffic flow in the Czech Republic is a pitfall. It mainly raises the question of how to verify pixel values (temperatures) and determine the range of the temperature interval for reliable detection of this category due to its currently small representation. Further research requires a better resolution of the thermal images. The proposed method of calculating thermal data is a promising first step to develop an automatic system for detecting vehicles in traffic flow, e.g., a joint vehicle recognition system and neural network. Both, the independence of detection on ambient conditions and categorization of passing vehicles according to the engine unit of the vehicles might be the advantages, compared to current detection systems. Recording other vehicle parts, especially the stern, is expected in the future. Moreover, detection of electric vehicles based on the absence of their exhaust system benefits from the temperature difference, compared to a combustion vehicle with an exhaust system.

6. Conclusions

The proposed approach shows a reliable detection of BEV, conventional and hybrid vehicles running on an internal combustion engine, and pure electric car. Exiting detectors cannot recognize these categories. The vehicle register does not deliver the necessary information about vehicle type. The main benefit of the proposed method is categorizing vehicles in real time. The use of detection is offered to the development of electric vehicles (EVs), especially in specific areas, such as historical city centers, etc. The use of reliable detection of BEV can be considered for various purposes, as mentioned in the introductory part, especially for toll systems. The thermal detection using methods of artificial neural networks can contribute to the further development of electromobility in cities, for instance, parking occupancy next to public charge points and charging stations. The described detection method has a wide potential in the concept of smart city, automatic diagnostics of vehicles in operation, vehicle classification, BEV preference allocation in traffic management in central urban areas, etc. It can contribute to safety in the city. Furthermore, thermal camera detection can address the priority of more occupied vehicles due to its ability to detect vehicle occupancy by passengers accurately. It can prevent fire ignition by detecting faults and overheating some vehicle parts in time. Thermal detection can be applied in tunnel technologies, additionally for checking diagnostic and condition prediction systems. The presented results offer the high potential of these technologies for electromobility, transportation, diagnostics, smart applications, their control, applicability, and further improvement [35].