Fuzzy Fusion of Stereo Vision, Odometer, and GPS for Tracking Land Vehicles

Abstract

The incorporation of high precision vehicle positioning systems has been demanded by the autonomous electric vehicle (AEV) industry. For this reason, research on visual odometry (VO) and Artificial Intelligence (AI) to reduce positioning errors automatically has become essential in this field. In this work, a new method to reduce the error in the absolute location of AEV using fuzzy logic (FL) is presented. The cooperative data fusion of GPS, odometer, and stereo camera signals is then performed to improve the estimation of AEV localization. Although the most important challenge of this work focuses on the reduction in the odometry error in the vehicle, the defiance of synchrony and the information fusion of sources of different nature is solved. This research is integrated by three phases: data acquisition, data fusion, and statistical evaluation. The first one is data acquisition by using an odometer, a GPS, and a ZED camera in AVE’s trajectories. The second one is the data analysis and fuzzy fusion design using the MatLab 2019^® fuzzy logic toolbox. The last is the statistical evaluation of the positioning error of the different sensors. According to the obtained results, the proposed model with the lowest error is that which uses all sensors as input (stereo camera, odometer, and GPS). It can be highlighted that the best proposed model manages to reduce the positioning mean absolute error (MAE) up to 25% with respect to the state of the art.

1. Introduction

A fundamental module of autonomous land vehicle navigation systems is absolute location, which in several works consists of a combination of data from different sensors including the GPS sensor [1]. It should be noted that high precision is required in the positioning of the vehicle to provide optimal security service. Recently, high precision positioning systems with an error of a few centimeters have been developed using cameras, inertial measurement unit (IMU), and light detection and ranging (LiDAR), among other methods [2,3,4,5,6]. The fundamental goal of data fusion is to obtain a lower probability of estimation error and greater reliability by using data from multiple sources. They are defined as complementary when the information provided by the input sources represents different parts of the scene and therefore could be used to obtain more complete global information. There is also redundant data fusion, which is used when two or more input sources provide information about the same objective and that information can be used to increase confidence, as well as a cooperative one when the information provided is combined with new information that is usually more complex than the original information [7,8]. In this sense, stereo visual odometry (SVO) sensors have become crucial for their ability to estimate camera movement based on images. SVO uses a relative scale to estimate the position and orientation of an agent using the images of one or more cameras attached to it. The SVO technique has been applied in the development of different vehicles such as autonomous vehicles, unmanned aerial vehicles, and submarines. Indeed, SVO was used to supplement the odometry of the wheel in the first exploration missions to Mars [9,10]. However, measurement noise and miscalibration induce an error (drift) in the estimated position, causing the heading reconstruction to deviate from the actual vehicle path. Slipping of the vehicle’s wheels in the sand caused the wheel odometry to malfunction, requiring a more accurate way to measure movement. For instance, estimating the relative displacement of the observed environment between images provides much more accurate results. Several works focused on various areas that are relevant in the image aspect. Such is the case of the topic of movement [9,11,12,13], in which the movement between two images of an environmental region is estimated by analyzing the relative displacements of the camera and the correspondences of characteristics present in the environment using various methods. On the other hand, there are studies focused on techniques for estimating depth from image [14,15,16,17,18,19,20], ranging from medical applications to applications of convolutional neural networks for robots. Several studies of embedded navigation systems are documented in the literature, showing how fuzzy systems can achieve this interaction by reducing error. For example, in [21], a study on an adaptive Kalman filter design based on fuzzy innovation is presented for the improvement of vehicle positioning performance in a dense urban environment. A fuzzy system is proposed to adaptively apply the covariance of the position precision, dilution measurement noise (DMN), the number of satellites that can be received, and the difference between the predicted position and the measurement position. In addition, the heading is checked and the covariance of the measurement noise of the inertial measurement unit and the on-board diagnostics (OBD) is applied based on the driving state of the vehicle. The proposed method corrects the MEMS-IMU yaw rate with the steering angle value obtained from the sensor on the vehicle and replaces the estimated MEMS-IMU speed by using the speed calculated from the 4-wheel revolutions per minute (RPM). This reduces the estimation error in the position, speed, and attitude of the MEMS-IMU.

The fuzzy inference has proven its power to compensate for sensor measurements and deliver a reference consistent output that helps to describe the measurement error. In [22], through intelligent fuzzy implications, an adaptive integration scheme is developed for the inertial navigation system (INS) with the global positioning system (GPS). Using knowledge-based fuzzy Mamdani-type inference, the inertial measurements comprising the outputs of the gyroscopes and accelerometers are examined to detect the level of vehicle maneuvering. Consequently, appropriate weighting coefficients are obtained to combine the INS/GPS and the attitude-heading reference system. In [23], a predictor-corrector guide method based on fuzzy logic is developed. The correction system is based on two fuzzy controllers, which correct longitudinal movement and lateral movement synergistically. The flight range error is eliminated by correcting the magnitude of the bank angle. The altitude error is eliminated by correcting the angle of attack. The lateral error is eliminated by regulating the inversion time of the bank angle. Compared to the traditional corrector based on Newton–Raphson iteration, the developed method only needs a single path prediction in a correction cycle, which is favorable for calculations. Furthermore, longitudinal movement and lateral movement are synergistically corrected in the predictor-corrector, which results in a robust and flexible method. Another important work is [24]. In this work they propose a simple method and the corresponding algorithm to build fuzzy logic systems to linearize sensors with a specified error in each interval of the measured parameter. The method has the advantage of requiring the same amount of memory as the piecewise linear approximation while the error decreases. Finding the FLS approximation is computationally only slightly more intensive than finding the optimal approximation piecemeal. The method is suboptimal in the sense that there are better solutions in terms of FLS for linearization but determining and effectively calculating these functions is computationally more intensive. In [25], the capacities of a routine, based on fuzzy logic, to elaborate a set of data from a CMM (Coordinate Measurement Machine) are analyzed. They show how to obtain, during hole measurement, the best measurement, so that the approximation error is minimized. In [26], a method based on fuzzy systems is considered for the correction of errors in Coriolis mass flowmeters through a two-phase fluid. The primary error-correcting capability of this technique is specified by testing through online experimental data. One more benefit of the suggested method is its small cost relative to other methods that require different hardware for error correction. In [22], a nonlinear correction algorithm for 2D-PSD based on a fuzzy neural network was investigated and implemented to reduce non-linear errors for two-dimensional position-sensitive detectors. Using this fuzzy neural network-based nonlinear correction algorithm, non-linear errors are reduced from ±0.4 mm to ±0.15 mm.

In this work, fuzzy fusion system data are designed, evaluated, and compared. This merges the data from three sensors: an odometer, a ZED camera, and a GPS module. Using the Takagi-Sugeno type fuzzy inference systems, which have been widely used in various industrial processes for a long time, this research takes advantage of the grand development in this technology. In recent years, due to the appearance of new electronic technologies, it has been decided to merge the information from different modern sensors. The deeper challenges of merging multiple sources lie in the nature of the information and its synchrony. This leads to highlighting some important aspects, such as the characteristics of the information, the representation of the information provided by the sources, and the formulation of the instructions on how to merge the information provided. We investigate the use of ANFIS to train a Takagi-Sugeno type fuzzy system with different membership functions at the inputs. Furthermore, in the method, we explain the architecture that reduces the error with respect to the actual literature [20,23].

In the following sections, the way in which the aforementioned research was developed is explained in detail. The article is organized as follows: Section 2 presents the Materials and Methods that include the general structure of fuzzy tracking system, the signals acquisition, fuzzy fusion, and the fuzzy system stages (fuzzification, inference, and defuzzification). In Section 3, the results of three different fuzzy fusion architectures are shown. Subsequently, in Section 4 of the discussion, the results are analyzed and compared to finally show the conclusions in Section 5.

2. Materials and Methods

The experimentation platform is an AEV developed in the TecNM autotronics laboratory in Celaya. Figure 1 shows the AEV and its sensors used in this experimental work: the odometer, GPS, and the stereo camera. The first is the FC-51 sensor, which is placed on the motor’s shaft to measure displacement from the number of revolutions (pulses) made by the engine and interpreted by a Raspberry Pi 3 mini computer. The second one is the GlobalTop FGPMMOPA6H, which uses the MediaTeK GPS chipset MT3339, and it can send position data over UART. The third is a ZED (StereoLabs, Orsay, France) camera, which allows one to perform stereoscopic visual odometry through the software included in it.

The measurement data were divided into two sets; these are the training and validation sets. The first used 70% of the measurements and the rest are for validation. To know the severity of the problem to be solved a priori, its particular deviations were calculated. The mean square error (MSE) of the odometer, GPS, and ZED are 448.56, 327.25, and 249.84, respectively. Figure 2 shows the route that was used to take the displacement data measured by the sensors of the electric vehicle shown in Figure 1.

The path was traveled in a closed circuit in a residential area. The total distance was 477.44 m. Distance traveled data were acquired with three sensors; these were odometer, GPS, and a ZED camera. Moreover, the trajectory was measured every meter with a tape measure.

2.1. General Structure of Fuzzy Tracking System

In Figure 3, the proposed method to estimate the position of the AEV is shown. The upper part of the diagram shows the three proposed sensors: the GPS, the infrared odometer, and the ZED camera. It also shows the nature of the data provided by the sensors; the GPS provides the latitude and longitude of the absolute position of the vehicle on the earth (in degrees), the infrared odometer provides the progress that the rear wheels of the AEV traveled (in meters), and the ZED stereoscopic camera provides an estimate of the AEV’s progress (in meters). The second step consists of the fusion of signals; here, it was important to analyze the data of different natures and synchronize the information at different sampling frequencies, as well as design the TS fuzzy system with different membership functions to the inputs to find the best option to reduce the estimation error with respect to the reference. This was automated using the Python programming language on a Raspberry Pi 3 (with a Broadcom BCM2837 64-bit 1.2 GHz processor and 1 GB of RAM). The design of the fuzzy system was developed with the support of ANFIS-Matlab 2019.

2.2. Signals Acquisition

The odometer system is an electronic optical sensor attached to the vehicle’s motor shaft. One revolution of the rotor corresponds to 0.67 m of forward advance. The optical sensor sends a signal for each turn to a Raspberry Pi 3. This information is stored in a count accumulator which corresponds to the displacement traveled. On the other hand, the latitude and longitude data are acquired by the GPS module and sent by using UART to a Raspberry Pi 3. The ZED stereoscopic camera is used to acquire images from the environment, and it estimates the translational and rotational movements of the terrestrial vehicle. Its estimation is also sent via USB communication to the Raspberry Pi 3. This camera is manufactured by the StereoLabs Company and uses the software tools provided by the manufacturer on their website. The camera can be operated with different resolutions and acquisition speed. In this work, the specifications were: 1280 × 720 resolution at 60 fps, the depth range that the camera has (0.3–25 m), and a viewing range: 90° (H) × 60° (V) × 100°, which was considered (D) max. The information generated by the three sensors is recorded in Raspberry Pi 3.

Once the measurements were obtained, the data from the three sensors was plotted and compared. Figure 4 shows the measurements made by each of the sensors. The odometer was not plotted because it only measures the forward or backward displacement relative to the AEV rear axle. It can be shown that there are differences in route reconstruction when using only the GPS or ZED camera compared to the actual route. It can also be noted that they both have a major error. The reference was drawn using QGIS software and it gives us the possibility to know the GPS coordinates of routes on real maps.

2.3. Fuzzy Fusion

Fuzzy logic is a discipline of Artificial Intelligence (AI) that is a generalization of classical logic. This has the advantage of using multiple degrees of membership between 0 and 1. Fuzzy inference systems (FIS) are used to reduce the uncertainty in position measurements. These have two fundamental parts, which are fuzzification and inference. The first aims to convert a numeric variable to linguistic variables. The second is made up of several linguistic rules that relate the input and output variables of the system. The Takagi-Sugeno fuzzy model is characterized by using linear functions of the inputs. Figure 5 shows the FIS proposed for the estimation of the ANFISedit position with MatLab 2019 [27,28,29].

In

Table 1. Fuzzy models proposed to estimate the distance.

Models	Input	MF	Outputs	MSE (5 Epochs)
1	Odometer, GPS X, GPS Y, ZED X, and ZED Y	20	X (m) and Y (m)	0.0335
2	Odometer, ZED X, and ZED Y	12	X (m) and Y (m)	0.1042
3	Odometer, GPS X, and GPS Y	12	X (m) and Y (m)	0.1382
4	ZED X and ZED Y	12	X (m) and Y (m)	0.0765

, several FIS (5 epochs training) proposed models for determining the distance traveled by the vehicle are shown. All models were trained with 70% of the measurements, with the MatLab ANFISedit module. The outputs of the fuzzy systems were the measurements made with a tape measure on the path traveled. Model 1 used the odometer, GPS, and ZED measurements as input, and its training error was 0.0335. For Model 2, it was the odometer and ZED measurements, and its training error was 0.1042. Model 3 used the odometer and GPS measurements as input, its training error was 0.1382. Model 4 used as input the positions of the GPS and ZED. Models 2 to 4 used 6, 6, and 8 membership functions, respectively, and in the case of model 1 there were 10; it can be noted that the smallest error is obtained by including the three sensor measurements in the fuzzy system.

2.4. Fuzzification

The fuzzification process consists of converting a real variable into a degree of belonging that quantifies the degree of possession toward its corresponding linguistic variable. To find and decide which ones, and the membership functions they would use, a search process supported by ANFIS was carried out. Different architectures were tested while their respective failures were recorded, which are shown in the following

Table 2. Proposed architecture errors.

Number of Inputs	Type of Membership Functions	MSE Latitude	MSE Longitude
6	trimf	0.001267	0.000634
6	trapmf	0.004761	0.000482
6	gbellmf	0.001637	0.000714
6	gaussmf	0.002035	0.000304
6	gauss2mf	0.022466	0.001828
6	pimf	0.02704	0.000729
6	dsigmf	0.012521	0.006350
6	psigmf	0.03409	0.001283

. As can be seen, the architecture with the least error for latitude is the triangular type. In the case of longitude, the option with the smallest error is that which uses Gaussian type functions.

According to the research carried out, summarized in Table 2, it was found that the best option for this problem is to use four triangular membership functions for latitude and four Gaussian membership functions for longitude with their respective linguistic variables (low, medium low, medium high, and high) shown in

Table 3. Triangular membership functions’ parameters.

Triangular Memberships
Infrared Odometer	GPS Latitude	Zed X
LOX = (−14952.15, −1.5131 × 10−7, 14952.15)	LGX = ([−15243.013, −11432.26, −7621.5)	LZX = (−15243.013, −11432.26, −7621.507)
LMOX = (−1.795 × 10−7, 14952.15, 29904.3)	LMGX = (−11432.259, −7621.507, −3810.753)	LMZX = (−11432.259, −7621.507, −3810.753)
HMOX = (14952.15, 29904.299, 44856.449)	HMGX = (−7621.507, −3810.753, −1.638 × 10−8)	HMZX = (−7621.507, −3810.753, −1.6386 × 10−8)
HOX = (29904.299, 44856.449, 59808.6)	HGX = (−3810.753, −4.8414 × 10−8, 3810.753)	HZX = (−3810.753, −4.841 × 10−8, 3810.753)

and

Table 4. Gaussian membership functions’ parameters.

Gaussian
Infrared Odometer	GPS Longitude	Zed Y
LOX = (6349.59, 0)	LGY = (1350.42, −699.77)	LZY = (38.31, −270.69)
LMOX = (6349.59, 14952.15)	LMGY = (1350.42, 2480.24)	LMZY = (38.31, −180.45)
HMOX = (6349.59, 29904.3)	HMGY = (1350.42, 5660.25)	HMZY = (38.31, −90.22)
HOX = (6349.59, 44856.45)	HGY = (1350.42, 8840.26)	HZY = (38.31, 0.012)

. The parameters for the triangular functions are the left vertex, center, and right vertex, assuming that the left and right vertices output zero whereas the center vertices have the unit as output (Table 3). On the other hand, the Gaussian parameters are the mean and the standard deviation, which are also listed in the following Table 4.

The membership functions used for latitude are of the triangular type and for longitude they are of the Gaussian type, as shown in Figure 6a–f. In these subfigures it is possible to appreciate the shape, the parameters, and the number of membership functions for each input.

2.5. Inference

The set of rules are of the if-else type with the AND operator; this was implemented with the scalar product between arguments. The result was delivered to each output variable corresponding to the respective output weights, $W_{xi}$ and $W_{yi}$. Two rules of the 64 inference rules for each of the two fuzzy systems FLX and FLY are shown below. In Appendix A, all the rules and their respective weights are listed.

• if (odometer is LOX) and (GPSX is LGX) and (ZEDX is LOZX) then (W_x1)
• if (odometer is LOX) and (GPSY is LGY) and (ZEDY is LZY) then (W_y1)

2.6. Defuzzification

Defuzzification was carried out using Equations (1) and (2), which used 128 inference rules. Figure 7 shows the Takagi-Sugeno fuzzy model, where Zi represents the weight of the output fuzzy rule, Wi is the weight of the membership function, and N the number of fuzzy rules [26,27].

$$F_{Lx}=\frac{W_{x1}\times Z_{x1}+W_{x2}\times Z_{x2}+W_{x3}\times Z_{x3}\dots +W_{xN}\times Z_{xN}}{W_{x1}+W_{x2}+W_{x3}\dots +W_{xN}}$$

(1)

$$F_{LY}=\frac{W_{y1}\times Z_{y1}+W_{y2}\times Z_{y2}+W_{y3}\times Z_{y3}\dots +W_{yN}\times Z_{yN}}{W_{y1}+W_{y2}+W_{y3}\dots +W_{yN}}$$

(2)

In order to observe the response of the fuzzy fusion system with five inputs and two outputs, the graphs between two inputs and one output were constructed and thus show the response of the complete system as illustrated in Figure 8. This gives us information on the work that the fuzzy system does to merge the information from the sensors and deliver the requested results.

3. Results

In Figure 9, the positioning responses of the three proposed fuzzy logic (FL) systems are shown. The result of fashioning the data by using the GPS sensor and the ZED stereo camera is shown in blue. In green is the trajectory obtained using an FL with the odometer and the GPS as inputs. The FL system which fuses the odometer, the ZED camera, and the GPS information is plotted in red. Finally, the ground truth or reference was drawn in black using the QGIS software. It can be observed qualitatively in the zoom of Figure 9 that the approximation in red color (Odometer-GPS-ZED) is closer to the reference, especially in rotations or direction changes. From a global point of view, the difference between the three fuzzy fusion systems is almost imperceptible. A fine analysis of these differences was, however, performed by calculating the error for each output. Figure 10 shows the results of that analysis for the best fuzzy fusion system.

Statistical Evaluation

The statistical evaluation was performed using the mean absolute error (MAE) metric. This has the advantage of identifying the outliers predicted by the system. Figure 10 shows the momentary absolute errors of the fuzzy system during the test run. Error peaks of 3.5 m can be observed, although their (MAE) is 0.128 m. This error is caused by predicting an atypical trajectory. This allows us to see the trend in both errors simultaneously. In both cases it is a small error, but the latitude response is affected to a greater extent in rotations (Sample 282) and in drastic speed changes (in Sample 800 of the graph).

4. Discussion

Our method showed a lower error rate than actual methods in the literature in trajectory reconstruction by using the data fusion from the GPS, odometer, and ZED camera. The use of fuzzy systems allows linearizing the combined sensor response; this coincides with the results reported by [23]. The errors obtained by the fuzzy systems were less than the 0.48 m and 15 m reported in [20,23], respectively. In

Table 5. Comparative of models to estimate the distance.

System	Technologies	Average Error
Woo, Yang, and Seo [20]	GPS and IMU	0.48 m
X. Wang [23]	GPS, IMU, and camera	15 m
System proposed	Odometer, GPS, and ZED camera	0.1265 m

, the average errors of the three systems are shown. Our proposed system has a smaller average error than the one reported in [20,23]. We can highlight that the responses of the fuzzy system are close to the reference trajectory.

The limitation shown by this configuration is the fact that the ZED camera becomes confused when facing the sun, whereas the GPS sends a greater error when there are obstacles between the sensor and the sky. Furthermore, the odometer in the configuration does not provide information about the laps.

The research carried out shows valuable information. The relative sensors, such as the odometer and the ZED camera, have a greater error in vehicle turns and a good precision in the straight course, whereas the global sensor (GPS) has a higher error in fast movements but a very good description at a global level. Each estimation of the position is complemented when a reasoning system adequately interprets the information.

5. Conclusions

In this work, a new fuzzy system was developed to merge the data from three sensors: an odometer, a ZED camera, and a GPS module. It allows for the estimation of the location with low error and competitive features with respect to other works of the state of the art. According to the results obtained, it can be concluded that the best proposed model presented the lowest MAE (0.1265 m). This is because the use of the fuzzy logic system complements the three different signals from the sensors used by the AEV location system. This architecture has the advantage of low cost and high precision. The proposed method can be extended for use in positioning systems of other types of autonomous vehicles, such as drones. Future work includes incorporating new sensors to improve the positioning of the AEV in different conditions and traffic paths.

Another adjunct challenge of this work at a technical level was the synchronization of the sensing time in the three sensors. This was solved by adding a mark to the file where the data of each source was saved. Among the most important limitations of this work is the speed of movement of the system, because the vehicle that was used reaches a maximum speed of 40 km/h.

We observe two directions in future research; the first is adding more sensors into the data fusion, such as: IMU, Electronic Compass, Lidar, among others. The second direction is to improve accuracy with fewer sensors for faster decision processing.