High-Speed Object Recognition Based on a Neuromorphic System

Abstract

Neuromorphic systems are bio-inspired and have the potential to break through the bottleneck of existing intelligent systems. This paper proposes a neuromorphic high-speed object recognition method based on DVS and SpiNNaker and implements a system in which an OR logic aggregation algorithm is used to acquire sufficient effective information and the asynchronous sparse computing mechanism of SNNs is exploited to reduce the computation. The experiment’s results show that the object detection rate of the designed system is more than 99% at the rotating speed of 900~2300 rpm; its response time is within $2.5 \mathrm{ms}$; and it requires 96.3% less computation than traditional recognition systems using the same scaled ANN.

1. Introduction

High-speed object recognition has been widely used in various fields, including autonomous driving, aerospace, sports photography and science experiments recording. While deep neural networks based on frames have shown impressive performance in high-speed object recognition [1,2,3], there are some limitations:

• The frame-based camera suffers from motion blur, which decreases the accuracy of recognition;
• There is high power consumption due to high data throughput and a huge number of multiply-add operations;
• The computation of such a full matrix deep neural network takes a large amount of time, which causes an unacceptable latency for high-speed recognition tasks.

An ideal high-speed object recognition system should have the characteristics of high accuracy, low power consumption and low latency. Since information is delivered as spikes, the third generation of neural networks, known as spiking neural networks (SNNs), have the sparsity of asynchronous computing. In comparison to artificial neural networks (ANNs), SNN’s utilization of bionic neurons allows the processing of information in the manner of the human brain, and it has the advantages of low power consumption and low latency, as well as a natural spatiotemporal information processing performance. However, SNNs can only benefit from their low power consumption and low latency if they are deployed on a neuromorphic computing platform [4,5,6]. At present, many neuromorphic computing platforms such as SpiNNaker [7,8], Loihi [4] and TrueNorth [9] have been developed. For data acquisition, high temporal resolution and low latency are required in high-speed scenarios. Dynamic vision sensor (DVS) [10], a neuromorphic vision sensor, has these characteristics, and the output is an asynchronous event stream rather than frames, which matches the asynchrony of SNN.

In this paper, we propose a novel high-speed object recognition method based on DVS and SNN and implement the corresponding recognition system using a neuromorphic computing platform. Experiments show that the system performs well at various speeds and is easy to transfer across different platforms. The main contributions of this paper can be summarized as follows:

• We present a high-speed object recognition system built on DVS and SpiNNaker that has a high level of adaptability to varied object speeds, high detection rate (more than 99%) and low latency (within $2.5 \mathrm{ms}$).
• On the DVS-based dataset, there is no degradation in accuracy for the SNN on SpiNNaker through comparative experiments, and the SNN running on SpiNNaker reduces the number of FLOPs by 96.3% compared to ANNs of the same scale.
• It is easy for the system to transfer to neuromorphic simulators without retraining.

The remainder of the paper is organized as follows: Section 2 introduces related work about DVS and SNN; Section 3 presents the neuromorphic high-speed object recognition method; Section 4 analyzes experiment results and evaluates the performance of the system; Section 5 goes to conclusions of the paper.

2. Related Work

DVS is a novel neuromorphic vision sensor and records light intensity changes, whose output is events. Compared with traditional frame-based cameras, DVS has advantages of high temporal resolution ($1 \mathsf{\mu }\mathrm{m}$), high dynamic range (over $120 \mathrm{dB}$), ultra-low power ($1 \mathrm{mW}$), and high pixel bandwidth ($\mathrm{kHz}$), which makes it more attractive than frame-based cameras for processing vision tasks such as image reconstruction [11,12,13], video frame interpolation [14,15], and object detection [16,17,18] in low light, high speed, and other extreme environments. Tulyakov, S. et al. [14] demonstrated the advantages of high temporal resolution and low latency of DVS to obtain better results in video frame interpolation applications. Perot, E. et al. [16] proposed a high-resolution large-scale dataset for object detection and demonstrated that DVS can perform comparable object detection tasks without image reconstruction.

Considering the compatibility between DVS and SNN which are bio-inspired, the study of combining them is an active research [19,20,21,22,23,24]. Liu, Q. et al. [21] extracted motion features from the event stream and built a hierarchical SNN for action recognition. Zhang, J. et al. [22] proposed an event-based object tracking method, in which SNN was introduced for tracking temporal cues. Viale, A. et al. [23] applied SNN, combined with DVS, in autonomous driving to recognize cars, and mapped the SNN on Loihi, which achieved both low latency and low power consumption.

When it comes to the experimental scene, some research employed rotating fans to simulate high-speed scenes and performed several computer vision tasks [25,26,27,28]. Zhao, J. et al. [25] and Zheng, Y. et al. [26] applied a neuromorphic vision sensor in image reconstruction of high-speed motion scenes. Zhang, S. et al. [28] presented an adaptive framework for detecting high-speed moving objects by combining DVS with traditional detection algorithms and showed that DVS could get sharp contours without motion blur, which was helpful for feature extraction and recognition. In this paper, we employ DVS to acquire data from a similar experimental scene and apply an SNN for recognition. By combining asynchrony and sparsity, we significantly reduce the latency and computation.

3. Methodology

3.1. Events Aggregation Algorithm

The output of DVS is an event stream, and each event can be represented by a 4-dimensional vector $\left(x,y,t,p\right)$, where $x,y$ denote the pixel coordinate; $t$ denotes the time stamp; the polarity parameter, $p$, can has a value of 0 or 1, with 1 signifying an increase in light intensity and 0 denoting a reduction. Figure 1 depicts the event stream.

In order to acquire enough information for the recognition, the events within a particular time interval are aggregated along time dimension to produce an event slice. This aggregation process can be expressed in Equation (1).

$$\{\begin{array}{l}{E}_i=\left\{e_{t^{\prime }}|t^{\prime }\in [\alpha i, \alpha \left(i+1\right))\right\}\hfill \\ E{S}_i=q\left(E_i\right)\hfill \end{array}$$

(1)

where $E_i$ is the set of events within the $i^{th}$ time interval; $e_{t^{\prime }}$ is the event with time stamp of $t^{\prime }$; $\alpha$ is the length of time interval; $E{S}_i$ is the $i^{th}$ event slice; $q(·)$ represents aggregation functions [29], which includes accumulation aggregation [30], non-polarity aggregation [31], OR logic aggregation [32], etc. All of these aggregation algorithms were compared in the experiment, and OR logic aggregation achieved the best performance due to its more stable features and less noise, which is shown in Figure 2. The whole process of OR logic aggregation can be expressed in Equation (2).

$$E{S}_i=sign\left({\displaystyle \sum e_{t^{\prime }}}\right), t^{\prime }\in [\alpha i, \alpha \left(i+1\right))$$

(2)

where $sign(·)$ is defined as: $sign\left(x\right)=1,\mathrm{if} x>0;sign\left(x\right)=0,\mathrm{if} x=0$. In summary, if a pixel produces at least one positive event in a certain time interval, the aggregation result on its position is set to 1, otherwise the value is 0. Since OR logic aggregation results in a binary image, it can be directly used as the input of SNN. In the experiment, we created a DVS-based dataset using the event slices aggregated by OR logic aggregation algorithm and evaluated its quality, which will be demonstrated in Section 4.2.

3.2. Spiking Neural Network Models

Various neuron models can be applied to construct SNNs. Hodgkin-Huxley model [33] is a biologically interpretable model, which uses variable conductance to simulate the opening and closing of ion channels on the membrane of biological neurons. However, since its dynamic equation involves multiple coupled differential equations, the computation is complex and not suitable for large-scale networks. Izhikevich’s model [34] greatly simplifies Hodgkin-Huxley’s model and reduces the number of parameters, where the total of twenty spike firing modes can be simulated with only four parameters. However, driven by nonlinear differential equations, the Izhikevich model is still computationally intensive. Therefore, the LIF [35] neuron model is used in this paper, which can be described by only one first-order linear differential equation shown in Equation (3).

$${\tau }_m\frac{dV\left(t\right)}{dt}=-\left(V\left(t\right)-V_{reset}\right)+X\left(t\right)$$

(3)

where $V\left(t\right)$ is the membrane voltage; $X\left(t\right)$ is the external input like the increment of voltage, which is provided by neurons in the previous layer or external stimulations; ${\tau }_m$ is called time constant, which is equal to the product of membrane resistance and capacitance, describing the decay rate of membrane voltage; $V_{reset}$ is the reset voltage.

The LIF neuron model needs to be manually reset after firing, and in practice, discretization for the dynamics is required. The charging, discharging, and resetting process in the discretized model can be expressed by Equation (4).

$$\{\begin{array}{l}H\left(n\right)=f\left(V\left(n-1\right), X\left(n\right)\right)\hfill \\ S\left(n\right)=\Theta \left(H\left(n\right)-V_{threshold}\right)\hfill \\ V\left(n\right)=H\left(n\right)\left(1-S\left(n\right)\right)+V_{reset}S\left(n\right)\hfill \end{array}$$

(4)

where $n$ is the time step index; $H\left(n\right)$ is the hidden state of neurons, which represents the instantaneous state of neurons before firing spikes; $f(·)$ is the state update function of neurons; $S(·)$ represents the spike train; $\Theta (·)$ is the step function; $V_{threshold}$ is the threshold voltage. The state update function of the LIF neuron used in this paper can be derived by Euler method to obtain its discrete form as shown in Equation (5).

$$f\left(V\left(n-1\right), X\left(n\right)\right)=V\left(n-1\right)+\frac{1}{{\tau }_m}\left(-\left(V\left(n-1\right)-V_{reset}\right)+X\left(n\right)\right)$$

(5)

For the network structure, this paper builds a fully connected SNN with two layers using the LIF neurons defined above. Since the event slices generated by OR logic aggregation has clear features, experiments show that a two-layer SNN without hidden layers is adequate to the recognition task. Furthermore, the network completes the computation in just one time step, which lowers latency. The firing and propagation of spikes in SNN is illustrated in Figure 3. When a sample is fed into the network, some specific input neurons will send spikes to all output neurons, then the output neuron that represents a particular class will fire a spike, thus completing the recognition.

In terms of network learning algorithm, the gradient-based backpropagation methods cannot be utilized in SNN effectively because of its non-differentiability. At present, ANN-to-SNN conversion methods [36,37,38,39] and surrogate gradient (SG) methods [40,41,42,43,44] can overcome the problem and achieve the prominent performance. Since the SG methods train SNNs with low latency, they are more suitable for high-speed object recognition scenarios. This paper applies continuous sigmoid function $S\approx \frac{1}{1+exp\left(-U\right)}$ [45], and makes use of its gradient to surrogate the gradient of step function in backpropagation. In addition, during the process of backpropagation, partial derivatives in time and space are computed separately.

3.3. Deployment on SpiNNaker

SpiNNaker is a successful general-purpose neuromorphic computing platform based on traditional hardware, which enables real-time, low-power and large-scale neural network simulations, as well as supporting more types of neuron models than dedicated neuromorphic computing platforms. Therefore, this paper employs it to deploy the above SNN.

Because neuron models provided in SpiNNaker are connected by synapses, which will convert spikes into electrical currents when transferring them, the computation of SNNs on SpiNNaker is different from that on CPU/GPU in Section 3.2. Furthermore, the currents will be affected by related parameters of neurons, hence the charging and discharging process in Equation (4) should be adjusted, resulting in Equation (6).

$$\{\begin{array}{l}I\left(n\right)=W^l{S}^{l-1}\left(n\right)\hfill \\ H\left(n\right)=I\left(n\right)R-e^{-\frac{\Delta t}{\tau }}\left(I\left(n\right)R+V_{reset}-V\left(n-1\right)\right)+V_{reset}\hfill \\ V\left(n\right)=H\left(n\right)\left(1-S^l\left(n\right)\right)+V_{reset}{S}^l\left(n\right)\hfill \end{array}$$

(6)

where $I\left(n\right)$ denotes the input current; $R$ denotes the membrane resistor; $W^l$ denotes weights of synapses in the $l^{th}$ layer; $\Delta t$ denotes the simulation time step; $S^l$ denotes the spike train fired by neurons in the $l^{th}$ layer; $\tau$ is a constant which is related to ${\tau }_m$ in Equation (3), and can be expressed in Equation (7).

$$\tau =1/\ln \left(\frac{{\tau }_m}{{\tau }_m-1}\right)$$

(7)

In summary, for data acquisition, DVS is used to acquire an asynchronous event stream, and the event stream will be aggregated to slices using OR logic aggregation. For recognition, a trained SNN is deployed on SpiNNaker, and the recognition result can be obtained by monitoring the spike fired by output neurons. The flow chart of the whole method is shown in Figure 4. In practice, in order to exclude invalid and redundant inputs, a selection strategy is introduced in data acquisition, which will be explained in Section 4.2.

4. Experiments and Analysis

4.1. Implementation of System

In this section, a neuromorphic high-speed object recognition system using the proposed method was implemented. The whole system is shown in Figure 5. The specific hardware used in the system is Prophesee Gen 4 and a SpiNNaker 101 machine. Through a USB link, the host configures and controls DVS by a data acquisition program. To configure relevant runtime parameters and download the trained SNN model onto the board, the host and SpiNNaker board communicate via Ethernet. The monitor program running on the host supervises the firing activity of output neurons in real time.

In the experiment, we built a fully connected SNN with 2 layers, including an input layer with 1600 LIF neurons (the size of input event slices is 40 × 40) and an output layer with 5 LIF neurons (5 classes). Each input neuron was one-to-one correspondent to an event position, and each output neuron represented a class. When an event slice was input into the SNN, only the target output neuron would fire a spike. Compared with traditional ANNs, the proposed network was independent of specific classification layers like argmax, softmax, etc., thus able to recognize objects much easier and faster. The experiment showed that some parameters of LIF neurons had impact on the performance of SNNs in both training and inference. The main parameters and models used in the system are shown in

Table 1. Parameter values and model types used in the experiment.

Parameter or Model	Value or Type
Constant $\tau$	$5 \mathrm{ms}$
$V_{reset}$	$0 \mathrm{mV}$
$V_{threshold}$	$12 \mathrm{mV}$
Synaptic dynamics model	delta-decay-type
Synaptic conduction model	current-type

.

4.2. Data Acquisition and Evaluation

Under laboratory conditions, this paper used a speed-adjustable fan as the high-speed scenario and recognized the letters on the blades, which is shown in Figure 6. There were 5 letters to recognize and the range of the rotating speed of the fan is from $900 \mathrm{rpm}$ to $2500 \mathrm{rpm}$. In order to only acquire one letter at a time, an ROI would be automatically set in advance, and only events in the area were taken into account. The steps to set the ROI are shown in Figure 6.

Data acquisition was implemented in two steps: aggregation and selection. For aggregation, the event stream within a certain interval was aggregated to generate an event slice by OR logic aggregation algorithm. The experiment showed that when time interval was set to $500 \mathrm{us}$, event slices were able to contain sufficient information of letters. Some aggregated event slices are visualized in Figure 7, where the pixel with value equal to 1, namely white pixel, is called “event pixel”. When a slice was fed into the SNN, event pixels would activate, instructing their corresponding neurons in the input layer to fire spikes.

For selection, since there were lots of useless event slices generated in aggregation step, a selection strategy was introduced to filter out invalid and redundant slices, which is illustrated in Figure 8. Experiments show that the number of event pixels in aggregated event slices is statistically significant under a fixed environment, as shown in Figure 8A. When letters rotate to the center of the ROI, the number of event pixels gradually increases; when letters rotate out of the ROI, the number of event pixels gradually decreases. In this paper, a Gaussian mixture model is used to fit the change in the number of event pixels, and then the event slices containing letters can be detected, as shown in Figure 8B. In order to avoid repetitive recognition of an identical letter in different locations, only slices with the letter in their center were selected to feed into the network after resizing to 40 × 40. In this manner, each letter would only be recognized once for each revolution of the fan.

Then a DVS-based dataset was created using the selected event slices, which totally contained 49,032 slices for 5 letters at different rotating speeds. In fact, the event slices were the trails of letter edges, and there was some noise if the event stream was aggregated directly.

4.3. Results Analysis

In the experiment, we evaluated the performance of the SNN-based system in terms of detection rate, response time and power consumption. Furthermore, in order to demonstrate the superiority of the neuromorphic system, we conducted some comparison experiments, including different platforms, datasets, and neural networks. In addition, we also verified the portability of the system on a simulator.

4.3.1. Performance Evaluation

The real-time performance of the designed system was evaluated under conditions similar to those of the DVS-based dataset. For the detection rate of the system, it is the product of the selection strategy success rate and the network recognition accuracy. After testing, when the fan speed was below $2300 \mathrm{rpm}$, our selection strategy worked well and the success rate remained above 99%; when the rotating speed exceeded $2300 \mathrm{rpm}$, the success rate of the selection strategy dropped rapidly, and we considered the strategy a failure. For recognition accuracy, since DVS could acquire clear letters at all speeds, the recognition accuracy of the SNN after training remained above 99% in the experiment, which is shown in

Table 2. Comparison of the SNN/ANNs on the DVS-based/RGB datasets.

	DVS+SN (SpiNNaker)	DVS+SNN (CPU)	RGB+FCNN	RGB+VGG16	DVS+FCNN
Accuracy	99.98%	99.98%	79.64%	100%	99.97%
Computation	<890	/	72,000	473 m	24,000

. Here we take the rotating speed of $1600 \mathrm{rpm}$ for example.

Response time mainly includes aggregation time and computing time. The aggregation time interval is $0.5 \mathrm{ms}$, as mentioned above, and the computing time is equal to $1 \mathrm{ms}$ or $2 \mathrm{ms}$, which is shown in detail in Figure 9.

In the experiment, computing time was measured through the internal clock in SpiNNaker with the highest precision of $1 \mathrm{ms}$. Theoretically, because SpiNNaker’s interlayer computation required one delay of $1 \mathrm{ms}$ and our network only had input and output layers, the computing time should always be $1 \mathrm{ms}$. However, as shown in Figure 9, there were some outliers with value of $2 \mathrm{ms}$. By analyzing the relationship between computing time and the number of event pixels that shows in Figure 10A–C, as well as conducting repetitive experiments shown in Figure 10D–F, we found that these outliers might occur when event pixels (i.e., input spikes) were more than 150. The reason was that, in SpiNNaker, when a neuron received a large number of spikes in one time step, it would receive these spikes in batches, which prolonged the computing time accordingly. However, not all inputs in the experiment with more than 150 event pixels produced outliers, because the clock’s precision was $1 \mathrm{ms}$ and if the starting point and the end point fell in adjacent time intervals while measuring the computing time, the final reading would be $1 \mathrm{ms}$.

Considering these outliers, the maximum response time was $2+0.5=2.5 \mathrm{ms}$, meeting the requirement of the fastest speed of $2500 \mathrm{rpm}$ in the experiment, i.e., completing recognition within $4.8 \mathrm{ms}$. And according to the configuration of the experiment, $2500 \mathrm{rpm}$ amounted to about $105 \mathrm{km}/\mathrm{h}$.

4.3.2. Comparisons and Analysis

We conducted comparative experiments in three aspects: deployment platforms, datasets and neural networks. For the deployment platform, we deployed the SNN on an i7-9700 CPU for recognition. For the dataset, we created an RGB dataset using an iPhone 13 pro max at 240 FPS in the same experimental scenario, which contained 21,430 samples. For neural networks, we built a fully connected neural network (FCNN) with the same scale as the SNN (1600 × 5) and introduced a VGG16 network as well. The comparison results are listed in Table 2.

For SNN, whether it was deployed on CPU or SpiNNaker, its recognition accuracy could reach almost 100%. However, due to its unique computing mechanism, SNN was incompatible with the ordinary CPU architecture, and the benefits of its asynchronous sparse computing in terms of power consumption could only be realized on a dedicated computing platform. In fact, in SpiNNaker, floating point numbers were emulated by fixed point numbers with the highest precision of $2^{-15}$, while that in CPU was $2^{-31}$. Hence there would be loss of numerical value in networks while deploying the trained SNN on SpiNNaker. But in the experiment, this loss did not affect the final recognition accuracy of the system, because our network had only two layers, and the loss would not accumulate in computation.

When the RGB camera was employed as the acquisition device, motion blur destroyed the original image features. For FCNN, its recognition accuracy could only reach about 80%, because its expressive power could not effectively extract and recognize features, even if the original features were very simple. The VGG16 had much stronger expressive power, so it could extract features and successfully recognize even images that heavily suffered from motion blur. However, it would involve a lot of computation, which required more than 473 million FLOPs. When the DVS was used as the acquisition device, event slices had a high image quality without motion blur, hence a 1600 × 5 FCNN was adequate to this recognition task, where 24,000 FLOPs were involved (8000 multiplications and 16,000 additions). For our solution of DVS+SNN (SpiNNaker), since only the input neurons corresponding to event pixels sent spikes, the computation of the network would decrease greatly. In the experiment, the maximum number of event pixels was 178, so there were up to 178 × 5 connections involved in computation with the amount of 890 FLOPs (only additions). Compared with the same scaled FCNN, the computation of the neuromorphic system declined by 96.3%, resulting in high energy efficiency.

4.3.3. Portability Analysis

Brian2 is a simulator for spiking neural networks. To verify the repeatability of our system, we reproduced the trained SNN on Brian2, and tested its recognition accuracy and computing time using the DVS-based dataset. The experiment result is shown in

Table 3. Recognition accuracy and computing time of the SNN on Brian2.

Rotating Speed	Recognition Accuracy	$\mathbf{Max}. \mathbf{Computing} \mathbf{Time}/\mathit{m}\mathit{s}$	$\mathbf{Min}. \mathbf{Computing} \mathbf{Time}/\mathit{m}\mathit{s}$	$\mathbf{Avg}. \mathbf{Computing} \mathbf{Time}/\mathit{m}\mathit{s}$
$1600 \mathrm{rpm}$	99.98%	452	102	156

. It could achieve the same recognition accuracy of the SNN on SpiNNaker, while its computing time is over 100 times longer because Brian2 simulates the computing of SNN based on the ordinary CPU architecture and is not comparable with neuromorphic computing platforms. During the simulation, the membrane voltages of five neurons in the output layer were also recorded, as shown in Figure 11. According to Figure 11, when the samples were fed into the SNN in sequence, the output neurons fired spikes in turn, which was consistent with the experimental expectations. This experiment showed that the system could transfer to an SNN simulator without retraining.

5. Conclusions

This paper presents a neuromorphic high-speed object recognition method and system, which applies OR logic aggregation algorithm to acquire enough effective information, and utilizes SNNs to reduce computation in recognition, where the SNN uses LIF neurons and is trained through a surrogate gradient method. The asynchrony and sparsity of DVS and SpiNNaker are used in the implementation of the system to achieve both high-speed data acquisition and computation.

In the experiment with rotating speed from $900 \mathrm{rpm}$ to $2300 \mathrm{rpm}$, the system achieves the detection rate of more than 99%, and the response time of each letter is within $2.5 \mathrm{ms}$. In addition, due to the computing mechanism of SNN and SpiNNaker, the number of FLOPs of the system were reduced by 96.3% compared with that using the same scaled ANN. Furthermore, our system can be directly transferred to other neuromorphic simulators without retraining. In future work, FPGA is to be considered as a micro-controller to further decrease data communication and power consumption.