Memory-Efficient Fixed-Length Representation of Synchronous Event Frames for Very-Low-Power Chip Integration

Abstract

The new event cameras are now widely used in many computer vision applications. Their high raw data bitrate levels require a more efficient fixed-length representation for low-bandwidth transmission from the event sensor to the processing chip. A novel low-complexity lossless compression framework is proposed for encoding the synchronous event frames (EFs) by introducing a novel memory-efficient fixed-length representation suitable for hardware implementation in the very-low-power (VLP) event-processing chip. A first contribution proposes an improved representation of the ternary frames using pixel-group frame partitioning and symbol remapping. Another contribution proposes a novel low-complexity memory-efficient fixed-length representation using multi-level lookup tables (LUTs). Complex experimental analysis is performed using a set of group-size configurations. For very-large group-size configurations, an improved representation is proposed using a mask-LUT structure. The experimental evaluation on a public dataset demonstrates that the proposed fixed-length coding framework provides at least two times the compression ratio relative to the raw EF representation and a close performance compared with variable-length video coding standards and variable-length state-of-the-art image codecs for lossless compression of ternary EFs generated at frequencies bellow one KHz. To our knowledge, the paper is the first to introduce a low-complexity memory-efficient fixed-length representation for lossless compression of synchronous EFs, suitable for integration into a VLP event-processing chip.

1. Introduction

A new type of sensor called an event camera was recently developed based on the new biomimetic technologies proposed in the neuromorphic engineering domain [1]. The event camera is bio-inspired by the human brain, as each pixel operates separately to mimic the biological neural system and performs simple tasks with small energy consumption. More exactly, in contrast to the conventional camera where all are operating simultaneously, the event camera sensor introduces a novel design where each pixel detects and reports independently only the changes (increased or deceased) of the incoming light intensity above a threshold or remains silent otherwise.

The event camera proposes a new paradigm shift for capturing visual data, where only the dynamic information of the scene is captured using a sequence (stream) of events that are triggered asynchronously, without capturing the unnecessary static information represented by the background or skyline. This gives the event camera some very important properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution, as the asynchronous events can be triggered individually at the smallest timestamp distance of one microsecond ($\mathsf{\mu }$s). Two types of sensors are now made available on the market: the dynamic vision sensor (DVS) [2], which captures sequences of asynchronous events; and the dynamic and active-pixel vision sensor (DAVIS) [3], which adds a second camera to the DVS sensor, an active pixel sensor (APS) for capturing greyscale or color (RGB) frames.

In the computer vision domain, event cameras are now widely used as the solutions designed to process the RGB and event modalities already reached state-of-the-art performance for many applications such as deblurring [4], feature detection and tracking [5,6], optic flow estimation [7], 3D estimation [8], super-resolution [9], interpolation [10], visual odometry [11], and many others. For a comprehensive literature review of event-based applications in computer vision, we recommend the work presented in [12]. Since many of these upper-level applications are designed to process the event modality as an image rather than an asynchronous events sequence, the event data acquired by the DVS sensor is pre-processed, usually using an event-accumulation process to form a sequence of synchronous Event Frames (EFs), i.e., the EF contains a set of synchronous events having assigned the timestamp associated with the EF.

In general, an EF is represented using the ternary alphabet, ${\mathcal{A}}_3^{raw}=\{0,1,2\}$, where for each pixel position $(x,y)$, one of the following three possible symbols is assigned: $“0”$ to signal the position of an event with negative polarity (report a light intensity decrease), $“1”$ to signal the position of a non-event, and $“2”$ to signal the position of an event with positive polarity (report a light intensity increase). Please note that sometimes the representation of the first two symbols is swapped. Although ${\mathcal{A}}_3^{raw}$ is used to store the EF on the disk, one can note that in some cases, a ternary alphabet containing different symbols might be used for a better visual representation. E.g., ${\mathcal{A}}_3^p=\{-1,0,+1\}$ is preferred for signaling the negative and positive event polarity; ${\mathcal{A}}_3^{int}=\{0,127,255\}$ is preferred for having an 8-bit intensity-like representation and guarantees compatibility with traditional image and video codecs. In the DVS sensor representation, each event is stored using a package of 8 Bytes (B). While in the raw EF representation, each pixel contains a ternary symbol that is stored using 2 bits, i.e., an EF having a $W\times H$ pixel resolution sensor is stored using a total of $2\times H\times W$ bits.

The event data captured by the event sensor contains different distortions (e.g., noise, latency), which must be corrected by the sensor or the event-processing chip, i.e., on low-power chips with limited memory before it can be consumed by upper-level applications. Since the raw EF representation reaches high bitrate levels, a more efficient fixed-length representation is required for low-bandwidth transmission of the event data from the event sensor to the event-processing chip. One can note that such a method must satisfy several constraints. The codec must operate in a group of pixels as other algorithms are employed to correct the event data using a limited chip memory, i.e., the codec must provide a fixed-length representation to provide random access to any group of pixels by allocating an equal number of bits to encode each group. Note that in data compression, large coding gains are achieved by allocating a variable-length representation to each group of pixels, as the coding gain depends on the amount of data stored inside each group. The codec must have a low complexity so it can be hardware written, i.e., only simple computations are accepted. The codec must provide an efficient memory representation to justify the costs of hardware writing the lossless codec. Hence, in this paper, we propose a novel low-complexity lossless compression framework of synchronous EFs based on fixed-length coding, which is suitable for integration into very-low-power (VLP) processing chips.

In the literature, the event data compression problem remains understudied. Only a few coding solutions are proposed to either encode asynchronous event sequences [13,14] or synchronous EF sequences [15,16]. In [13], the authors propose to encode lossless asynchronous event sequences by exploiting the spatial and temporal characteristics of the event location information. The research was further extended in [14]. In [15], the Time Aggregation-based Lossless Video Encoding for Neuromorphic vision sensor data (TALVEN) algorithm is proposed, where an event-accumulation process is employed to generate EFs by concatenating the positive and negative polarity EF counts and form an EF sequence then compresses by the High-Efficiency Video Coding (HEVC) [17] standard. Finally, in [16], a lossy coding solution is proposed for the DAVIS sensor. One can note that these solutions are much too complex for hardware implementation in the event-processing chip with limited memory. These are performance–oriented compression solutions that can be integrated only in a system-on-a-chip (SoC) where enough computation power is available. Here, we propose a low-complexity–oriented lossless compression framework of EFs suitable for integration into low-power event signal processing (ESP) chips or VLP event-processing chips, i.e., inside the sensor, where the event data are constrained to have fixed-length representation.

In our prior work, we employ an event-accumulation process where each asynchronous event sequence is split into spatiotemporal neighborhoods of time interval $\Delta ,$ where each neighborhood generates an EF as follows: at each pixel position $(x,y)$, the polarity sum of the events triggered in the time interval $\Delta$ is computed so that $(x,y)$ is set as the sum’s sign, see Figure 1. In our prior work [18], we proposed a context-based lossless image codec for encoding a stack of up to eight EFs using two data structures: an event map image (EMI) that stores the event spatial information; and the concatenated polarity vector (CPV) that stores the polarity information; a lossless codec suitable for integration into SoC. In our prior work [19], we proposed a low-complexity lossless coding framework by adapting the run-length encoding scheme and Elias coding for EF coding, suitable for integration into low-cost ESP chips. While in our prior work [20], we proposed a low-complexity lossless coding method for encoding the event data represented as asynchronous event sequences that employ only low-complexity coding techniques so that it is suitable for integration into low-cost ESP chips. Moreover, these solutions provide a variable-length representation, a strategy used by most state-of-the-art coding methods. In contrast, the goal of this work is to introduce a novel low-complexity lossless coding framework for fixed-length representation of EFs using multi-level look-up tables (LUTs) that is suitable for integration into VLP chips.

In summary, the novel contributions proposed in this paper are listed below.

(1)

A novel lossless compression framework for encoding synchronous EFs, suitable for hardware implementation in VLP event-processing chips; see Section 3.

(2)

A novel ternary frame representationmusing group partitioning and symbol remapping; see Section 3.1.

(3)

A novel memory-efficient low-complexity coding scheme for lossless compression of sparse images using a novel fixed-length representation based on multi-level LUTs; see Section 3.2.2.

(4)

An improved fixed-length representation using a mask-LUT structure for encoding EFs using very-large groups of pixels; see Section 3.2.3.

(5)

A codec that provides random access (RA) to any group of pixels using a fixed-length representation; see Section 3.3.

The remainder of this paper is organized as follows. Section 2 presents a discussion on state-of-the-art methods developed for the new event modality. Section 3 describes the proposed fixed-length coding framework. Section 4 presents the experimental evaluation of the proposed framework. Section 5 concludes this work.

2. State-of-the-Art Methods

Let us consider an event sensor having a resolution of $W\times H$ pixels. When the sensor detects a change in the incoming light intensity that is above a set threshold, an asynchronous event, denoted $e_i=(x_i,y_i,p_i,t_i),$ is triggered by encapsulating and transmitting 8 B of event data containing the following information: (i) the event spatial information, denoted $(x_i,y_i),$ where $x_i\in [1,H],$ and $y_i\in [1,W],$ corresponding to the pixel position where the change was detected; (ii) the event polarity information, denoted $p_i\in \{-1,1\},$ which signals the type of change using symbol $“-1”$ for a decrease and symbol $“1”$ for an increase in the incoming light intensity; and (iii) the event time information, $t_i,$ i.e., the timestamp when the change was detected. Let us denote the asynchronous event sequence capture by a DVS sensor [2] as ${\mathcal{S}}_{\mathcal{T}}={\left\{e_i\right\}}_{i=1,2,\dots ,N_e},$ where $N_e$ events were triggered over the time period $\mathcal{T}$ $\mathsf{\mu }$s.

One can note that the processing steps in event-based computer vision solutions are significantly different than in the case of conventional color cameras. In the event-signal processing framework, an important decision is represented by the selection of the event representation. ${\mathcal{S}}_{\mathcal{T}}$ is first extracted from the raw DVS data and then further transformed into the format interpretable by the following processing steps, which may differ from one application to another.

The section is organized as follows. Section 2.1 analysis the event representations proposed in the computer vision domain. Section 2.2 outlines the proposed state-of-the-art methods for lossless event data compression and analysis how to modify the traditional data compression codecs for lossless event data compression.

2.1. Event Data Representations in Computer Vision

The simplest event representation that gained a lot of popularity in recent years is spike processing, such as Spike Neural Networks (SNNs), which directly can process the asynchronous event sequences [22]. Another representation used in Computer Vision is the “image-like" representation, denoted here EF, as most of the upper applications prefer to consume the event data as an image rather than an asynchronous event sequence. One can note that the event data can be stored as a sequence of generated EFs in a lossless manner (no information is lost) only by setting $\Delta$ with the smallest time-window size, i.e., $\Delta =1$ $\mathsf{\mu }$s. Then the raw EF representation will require an impressive amount of space to store one second of captured event data, i.e., $\frac{H\times W}{4}$ MB (around $76.8$ GB). Note that the raw EF representation (2-bit per pixel) is more efficient than the raw sensor representation (8 B per event) only if more than $p_{\tau }=3.125\%$ of the positions in the generated EF sequence signal event positions, i.e., a capturing event density that the DVS sensor cannot reach using current technology. Therefore, $\Delta$ is usually set using values in the ms time range so that the generated EFs have a much higher matrix filling percentage than $p_{\tau },$ which results in losing some part of the raw captured information.

Different pre-processing strategies were proposed in the literature to quantize the event data and further process it in such a way that an upper-level application can achieve state-of-the-art performance. In [23], the event polarity sum frames are computed by summing the polarity of events within a given time window for real-time visual-inertial odometry. In [24], the count-based event-accumulation representation is proposed for steering prediction for self-driving cars, where the events within a temporal window are counted for each polarity type to generate two EFs. In [25], the distance surface representation is proposed for pixel motion estimation, where the generated image contains the spatial distance to the active pixel. In [26], the surface of active events representation is proposed for computing the visual flow, where the EF contains only the latest event triggered at each pixel position. In [27,28], the graph-based representation is proposed for object classification, where the events are transformed within a temporal window into a set of connected nodes. Although this representation is quite compact and efficient, the generated graphs can be computationally expensive. In [29], the voxel grid representation is proposed. In [30], the event spike tensor representation is proposed as a voxel grid variation. In [31], the time-ordered recent events representation is proposed for a wide range of applications (denoising, reconstruction, classification, pose estimation), where a first-in-first-out buffer is used to retain the most recent events at each location, while the others are ignored.

2.2. Event Data Representations in Lossless Compression

In the lossless compression domain, two event data representations, asynchronous event sequences, and event-accumulation EFs, were studied. In [13], the authors propose a solution for encoding the raw asynchronous event sequences by removing the redundancy of the spatial and temporal information using three strategies: adaptive macro-cube partitioning structure, the address-prior mode, and time-prior mode. An extension was proposed in [14] by introducing an event sequence octree-based cube partition and a flexible inter-cube prediction method based on motion estimation and motion compensation. In [32], the authors present a complex performance analysis of different lossless data compression algorithms employed to compete against the Spike Coding approach proposed in [14]. The study shows that several state-of-the-art traditional data compression codecs, such as the Lempel–Ziv–Markov chain algorithm (LZMA) [33] developed by Igor Pavlov and the dictionary-based codec Zeta Library (ZLIB) [34], provide a good coding performance. One can note that the comparison with [13] and [14] is not possible as their codec/source code is not made publicly available, and the PKU-DVS dataset [35] is partially made available only for academic research purpose.

The study in [32] was extended in [15] by introducing the TALVEN algorithm for encoding the count-accumulated EF sequences so that they can be further encoded by employing the HEVC codec [17]. For encoding event-accumulation EFs, in our prior work [18,19], we proposed to compare the coding performance with that of prior and current video coding standards, HEVC [17] and VVC [36], and of well-known state-of-the-art lossless image codecs, such as Context Adaptive Lossless Image Codec (CALIC) [37], and Free Lossless Image Format (FLIF) codec [38]. Experiments show that an improved performance is achieved when encoding an intensity-like event representation obtained using a symbol remapping technique where a set of five (consecutive) EFs is merged into a combined EF that contains (8-bit) intensity-like values.

In conclusion, the current coding solutions of event data, either based on event codecs [13,14,15] or traditional data/image/video codecs [17,33,34,36,37,38], provide competitive variable-length representations designed to optimize the codec’s performance. In contrast, in this work, we propose a fixed-length representation designed to achieve low complexity and fast RA to any group of pixels, suitable for integration into VLP chips.

3. Proposed Framework

Figure 2 presents a comparison between the requirements imposed for different compression strategies. In a left-to-right order, the first representation is the raw image representation, where no compression strategy is employed as all pixels are represented uncompressed in memory using a fixed-length representation of the maximum number of bits required to store each symbol in the alphabet, and which provides random access to any pixel. The first compression strategy is the fixed-length representation, where on top of the fixed-length and random-access requirements, the codec must provide a more efficient representation of the image in the memory, i.e., to store the image using a reduced number of bits compared with raw representation. Moreover, such compression methods usually propose to encode together a group of pixels (e.g., a $w\times h$ pixel-block) and must have a very low computational complexity so that it can be directly hardware written in the sensors. The second compression strategy is the variable-length representation with random access, where the coding performance is further improved by removing the fixed-length representation of each group of pixels and where a header is introduced to store additional information needed to provide random access to each group of pixels. The last compression strategy is storage-optimized, i.e., it is designed to provide the highest coding performance possible without imposing any constraints on algorithmic complexity, runtime, or memory requirements.

One can note that the fixed-length representation requirement is a very hard constraint and does not allow the solution to achieve the best performance. The fixed-length and variable-length representations offer the possibility to apply any pre-processing algorithm to a group of pixels while still storing in memory the entire package of data. In general, variable-length representations, also known as performance-oriented codecs, are optimized to achieve the best coding performance without imposing any constraints on complexity, runtime, or having access to groups of pixels.

Figure 3 presents what type of compression strategies can be used for each type of chip. E.g., the VLP chips may incorporate a lossless compression method as long as its complexity is very low so that it is possible to be written on hardware, and each group of pixels can be encoded separately so that they can be packetized together. The ESP chip may incorporate a lossless compression method to reduce the amount of data to be transmitted. Such a chip contains a limited memory of a few MB, and the codec must be simple enough to be hardware written. While in general, the SoC chip may incorporate any compression method as long as the SoC specifications allow the coding method to be run on the SoC. Finally, the data are transmitted from the SoC chip to be stored on the non-volatile memory of a mobile phone (e.g., memory card) or a computer (e.g., hard disk drive).

In this work, we propose a fixed-length representation framework suitable for integration into VLP chips, which offers the possibility to store more efficiently in memory the input image compared with the raw image representation. Figure 4 presents the proposed fixed-length representation framework for encoding the input raw image representation. The proposed strategy consists in first dividing the raw image into groups of pixels of $w\times h$ size, where each group is represented using a reduced set of ($N_t$) remapped symbols. The figure’s central part shows that the raw image can be represented as remapped volume having group-partition resolution and remapped-group-size depth. While the right part shows the proposed method is employed to store the remapped volume using two data structures: a matrix for storing using a fixed-length representation of an index and additional information and a set of one or more LUTs for storing the unique symbols combination used to remap the original raw data.

The section is organized as follows. Section 3.1 presents the group partitioning and symbol remapping processes. The remapped volume is used to generate the proposed fixed-length representation consisting of an index matrix and a set of one or more LUTs. Section 3.2 presents the proposed LUT-based Representations. Section 3.3 presents the final algorithmic details.

3.1. Group Partition and Symbol Remap

Let us denote $\mathbf{I}$ the input image of size $W\times H$ which stores using 2 bits a ternary symbol, $s\in \{0,1,2\},$ at each pixel position $(i_r,j_r),\forall i_r\in [1,H],j_r\in [1,W].$ In this work, we propose to partition $\mathbf{I}$ into $N_g=\frac{W}{h}\times \frac{H}{h}$ groups of pixels, where each group $G_{i_g{j}_g},\forall i_g\in [1,\frac{H}{h}],j_g\in [1,\frac{W}{w}],$ of size $w\times h$ contains $N=wh$ ternary symbols. Firstly, each group of pixels $G_{i_g{j}_g}$ is vectorized as follows:

$$V_{i_g{j}_g}^v=\left[s_0{s}_1\dots s_{N-1}\right].$$

(1)

Next, $V_{i_g{j}_g}^v$ is then zero-padded up to the closest multiple-of-five length such that the zero-padded vector can be split into $N_t=\lceil \frac{N}{5}\rceil$ subgroups of five ternary symbols, see Figure 5. Let us denote the zero-padded group vector as $V_{i_g{j}_g}^z,$ where

$$V_{i_g{j}_g}^z=\left[s_0{s}_1\dots s_{5N_t-1}\right]=\left[[s_0{s}_1\dots s_4]\dots [s_{5k}{s}_{5k+1}\dots s_{5k+4}]\dots [s_{5N_t-5}{s}_{5N_t-4}\dots s_{5N_t-1}]\right].$$

(2)

Each subgroup k of five ternary symbols, $[s_{5k}{s}_{5k+1}\dots s_{5k+4}],$ is then remapped into a symbol t in the alphabet $\{0,1,\dots ,3^5-1\}$ (represented using 8 bits), by employing the following symbol remapping formula:

$$t_k=\sum _{r=0}^4{3}^{4-r}{s}_{5k+r},k=0,1,\dots ,N_t.$$

(3)

Figure 5 shows that $V_{i_g{j}_g}^z$ is remapped as follows:

$$V_{i_g{j}_g}=\left[t_0{t}_1\dots t_{N_t-1}\right].$$

(4)

Furthermore, $V_{i_g{j}_g}$ can be further rearranged as a remapped unite volume of size $1\times 1\times N_t.$

In conclusion, the proposed fixed-length representation first stores the input image $\mathbf{I}$ as a remapped volume $V(i_g,j_g,t_k),\forall i_g\in [1,\frac{H}{h}],j_g\in [1,\frac{W}{w}],k\in [0,N_t],$ of size $\frac{W}{w}\times \frac{H}{h}\times N_t,$ as depicted in the middle part of Figure 4. One can note that the input image $\mathbf{I},$ having a raw image representation of $2WH$ bits, stored in memory using the proposed remapped volume representation, $\mathbf{V},$ using $8N_g{N}_t=8\frac{WH}{wh}\lceil \frac{wh}{5}\rceil ,$ i.e., the input memory size is reduced with a percentage of

$$p_{mr}=100-\frac{400}{wh}⌈\frac{wh}{5}⌉(\%).$$

(5)

A maximum memory reduction of $p_{mr}=20\%$ (or a compression ratio of $1.25,$ see Section 4) is obtained when the pixel-group size, $N,$ is selected as a multiple-of-five number.

3.2. Proposed LUT-Based Representations

The proposed fixed-length representation is designed to encode each group $G_{i_g{j}_g}$ using an index value (represented on a specific number of bits) stored in the index matrix $\mathbf{M}.$ The main idea is to find all unique combinations of $N_t$ symbols found in any vector $V_{i_g{j}_g}$ and store them in memory using a novel LUT-based structure that continues to provide a fixed-length representation. Therefore, the remapped volume representation, $\mathbf{V},$ is represented using a fixed number of bits by the index matrix $\mathbf{M}$ and the LUT-based structure, see the right part of Figure 4, where $M(i_g,j_g)$ contains all the necessary information to extract $V_{i_g{j}_g}$ from the LUT-based structure. Several fixed-length representation solutions based on different levels of LUT indexing are proposed. A variable-length representation solution for storing the LUT-based structure is also studied.

Section 3.2.1, Section 3.2.2, and Section 3.2.3 present the proposed fixed-length representation solutions based on a single-level LUT, double-level LUTs, and multi-level LUT structures, respectively. Section 3.2.4 presents the variable-length representation solution for compressing the LUT-based structure.

3.2.1. Single-Level LUT Solution

The proposed single-level LUT solution, called 1L-LUT, introduces a simple fixed-length representation by storing the $N_{uc}$ unique combinations of $N_t$ symbols using a single LUT table structure, which requires

$${\mathcal{L}}_{LUT}^{1L}=8N_t{N}_{uc}\mathrm{bits}.$$

(6)

In such case, $M(i_g,j_g)$ stores an index $\kappa$ using $n_{\kappa }=\lceil lo{g}_2{N}_{uc}\rceil$ bits, where $V_{i_g{j}_g}$ is extracted from LUT as $V_{i_g{j}_g}=LUT\left(\kappa \right).$ The index matrix $\mathbf{M}$ is represented using

$${\mathcal{L}}_M^{1L}=N_g{n}_{\kappa }\mathrm{bits}.$$

(7)

The number of bits needed to represent $\mathbf{I}$ using the 1L-LUT solution is computed as follows:

$${\mathcal{L}}^{1L}={\mathcal{L}}_M^{1L}+{\mathcal{L}}_{LUT}^{1L}=N_g{n}_{\kappa }+8N_t{N}_{uc}\mathrm{bits}.$$

(8)

Figure 6 depicts the proposed 1L-LUT solution. One can note that the value $\kappa -1$ (instead of $\kappa$) is written in $M(i_g,j_g)$ using $n_{\kappa }$ bits as $\kappa \in \{1,2,\dots ,N_{uc}\}.$ For example, if $N_{uc}=4,$ then only two bits are needed to represent the index $\kappa =4$ instead of three ($4_{\left(10\right)}={100}_{\left(2\right)}$). ${\mathcal{L}}^{1L}$ depends on the group size, $N,$ as follows:

(i)

If N is too small, then $\mathbf{M}$ contains too much information, and $N_{uc}$ is too large as too many unique combinations are stored by LUT;

(ii)

If N is too large, then $\mathbf{M}$ contains too less information, and each line in LUT contains too much information as $N_t$ is too large.

An optimal $N^{*}$ provides the smallest memory representation for the 1L-LUT solution, however, its coding performance remains limited.

3.2.2. Double-Level LUT Solution

The proposed double-level LUT solution, called 2L-LUT, introduces a fixed-length representation where the $N_{uc}$ unique combinations of $N_t$ symbols are further divided into a set of $N_t$ LUTs, ${\left\{LU{T}_{\ell }\right\}}_{\ell =1:N_t}$, to achieve an improve compression performance and faster search of the unique combinations. The main idea is that $V_{i_g{j}_g}$ is first classified into $N_{uc}+1$ classes by counting the number of non-zero values as $\ell .$ 2L-LUT propose to created $N_t$ LUTs, where the ℓ-th LUT, denoted LUT${}_{\ell },$ stores $N_{uc}^{\ell }$ unique combinations of $N_t$ symbols found in any vector $V_{i_g{j}_g}.$ One can note that for $\ell =0$, no LUT is needed to be stored in memory as a single deterministic case can be found: all values are zero. In such case, $M(i_g,j_g)$ stores the LUT index, $\ell ,$ using $n_{\ell }$ bits, see Equation (9), and the index in the LUT, ${\kappa }_{\ell },$ using $n_{\kappa }^M$ bits, see Equation (10), i.e., using the maximum number of bits needed to represent the largest number of unique combinations in the set of LUTs. Hence, $M(i_g,j_g)$ stores ℓ using the first $n_{\ell }$ bits and ${\kappa }_{\ell }$ using the following $n_{\kappa }^M$ bits. $V_{i_g{j}_g}$ is extracted from the ℓ-th LUT as $V_{i_g{j}_g}=LU{T}_{\ell }\left({\kappa }_{\ell }\right).$ Therefore, $\mathbf{M}$ is stored by 2L-LUT using ${\mathcal{L}}_M^{2L}$ bits, see Equation (11).

$$n_{\ell }=\lceil lo{g}_2{N}_t\rceil$$

(9)

$$n_{\kappa }^M=\lceil {log}_2(max{\left\{N_{uc}^{\ell }\right\}}_{\ell =1:N_t})\rceil$$

(10)

$${\mathcal{L}}_M^{2L}=N_g(n_{\ell }+n_{\kappa }^M)$$

(11)

The set of $N_t$ LUTs, ${\left\{LU{T}_{\ell }\right\}}_{\ell =1:N_t}$, contains $N_{uc}^{\ell }={\sum }_{\ell =1:N_t}{N}_{uc}^{\ell }$ entries. 2L-LUT makes use of the information that $LU{T}_{\ell }$ (the ℓ-th LUT) contains ℓ non-zero symbols by introducing a mask of $N_t$ binary symbols, $b_{\ell },$ where the ℓ-th mask symbol is set by checking if the corresponding symbol $t_{\ell }$ is or not symbol $“0”$ in the alphabet, see Equation (12).

$$b_{\ell }=\left\{\begin{array}{cc}\hfill 0\hfill & \hfill \mathrm{if}{t}_{\ell }=0\hfill \\ \hfill 1\hfill & \hfill \mathrm{if}{t}_{\ell }\ne 0\hfill \end{array}\right.,\ell =0,1,\dots ,N_t-1$$

(12)

The ℓ non-zero symbols are stored using $8\ell$ bits. Hence, 2L-LUT stores LUT${}_{\ell }$ in memory using ${\mathcal{L}}_{LU{T}_{\ell }}^{2L}$ bits, see Equation (13), and $\mathbf{I}$ using ${\mathcal{L}}^{2L}$ bits, see Equation (14).

$${\mathcal{L}}_{LU{T}_{\ell }}^{2L}=(N_t+8\ell )N_{uc}^{\ell }$$

(13)

$${\mathcal{L}}^{2L}=N_g(n_{\ell }+n_{\kappa }^M)+\sum _{\ell =1}^{N_t}(N_t+8\ell )N_{uc}^{\ell }$$

(14)

Figure 7 depicts the proposed 2L-LUT solution. Note that $n_{\kappa }^M$ can be computed only after populating the entire set of LUTs with unique combinations of $N_t$ symbols. Therefore, the following bit-cleaner procedure, BitCleaner($n_{\kappa }^M$), is employed to guarantee a single pass encoding of $\mathbf{I}.$

BitCleaner($n_{\kappa }^M$) procedure:

(i) 

Compute $n_w=\lceil {log}_2{N}_g\rceil$ as $N_{uc}^{\ell }\le N_g,$ i.e., in the extreme case each group contains a unique combination of $N_t$ symbols.

(e) 

Employ 2L-LUT and write each value ${\kappa }_{\ell }-1$ using $n_w$ bits instead of $n_{\kappa }^M$ bits.

(b1) 

Compute $n_{\kappa }^M$ using Equation (10) and ${\left\{N_{uc}^{\ell }\right\}}_{\ell =1:N_t}.$

(b2) 

Remove the unnecessary $n_w-n_{\kappa }^M$ bits, $n_{\ell }+n_{\kappa }^M+1:n_{\ell }+n_w,$ from the representation of each index value $M(i_g,j_g).$

In the encoding algorithm, one can note that step (i) is employed in the initialization stage, step (e) in the encoding stage; and steps (b1)–(b2) in the final processing stage of the final bitstream, ${\mathcal{B}}_{\mathcal{M}},$ generated for encoding the index matrix.

A similar procedure was developed to post-process the bitstream and remove the unnecessary bits used in the signaling of ℓ for the large group size case, where not all LUTs are used in the proposed LUT-based representation, i.e., only the first few LUTs are used as, for example, $LU{T}_{N_t}$ has the lowest probability to be used included in the LUT set. Therefore, the number of bits needed to represent ℓ is the number of bits needed to represent ${\ell }^{*},$ the index of the last LUT in the set for which $N_{uc}^{{\ell }^{*}}\ne 0.$ The bit-cleaner procedure, BitCleaner($n_{\ell }^{*}$), is employed.

BitCleaner($n_{\ell }^{*}$) procedure:

(i)

Compute $n_{\ell }=\lceil {log}_2\frac{HW}{hw}\rceil .$

(e)

Employ 2L-LUT and write each value $\ell -1$ using $n_{\ell }$ bits.

(b1)

Compute $n_{\ell }^{*}=\lceil {log}_2\left(N_{uc}^{{\ell }^{*}}\right)\rceil .$

(b2)

Remove the extra $n_{\ell }-n_{\ell }^{*}$ bits, $n_{\ell }^{*}+1:n_{\ell },$ from the representation of each $M(i_g,j_g).$

Similarly, step (i) is employed in the initialization stage, (e) in the encoding stage; and steps (b1)–(b2) in the final processing stage of ${\mathcal{B}}_{\mathcal{M}}.$

3.2.3. Multi-Level LUT Solution

When the selected group size is very large, for example, for $N_t\ge 150,$ the memory allocated to store the LUT-based structure of 2L-LUT contains a lot of redundancy. One can note that in such a case, the number of unique combinations in the set of LUTs is reduced, and the unique combinations are more populated with the symbol $“0”,$ the most frequent symbol in the alphabet. In this work, we propose to modify 2L-LUT for very large block sizes further and introduce the multi-level LUT solution called ML-LUT. ML-LUT introduces an extra LUT compared with 2L-LUT, called LUT mask and denoted $LU{T}_M,$ to store a mask associated with each LUT in ${\left\{LU{T}_{\ell }\right\}}_{\ell =1:N_t},$ i.e., line ℓ in $LU{T}_M$ is associated with $LU{T}_{\ell }.$ Therefore, the position k in line $\ell ,$$LU{T}_M(\ell ,k),$ is set by checking if all the unique combinations stored by $LU{T}_{\ell }$ contain a symbol $“0”$ on the k-th position, i.e., $LU{T}_{\ell }(q,k)=0,\forall q=1,2,\dots ,N_{uc}^{\ell },$ see Equation (15). Let us denote $\pi$ as the number of lines in $LU{T}_M,$ and ${\mathcal{L}}_{Mask}$ as the number of bits required to store $LU{T}_M,$ computed using Equation (16).

$$LU{T}_M(\ell ,k)=\left\{\begin{array}{cc}\hfill 1\hfill & \hfill \mathrm{if}LU{T}_{\ell }(q,k)=0,\forall q=1,2,\dots ,N_{uc}^{\ell }\hfill \\ \hfill 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.,\ell ,k=1,2,\dots ,N_t$$

(15)

$${\mathcal{L}}_{Mask}^{ML}=N_t\pi$$

(16)

Figure 8 depicts the proposed ML-LUT solution. One can note that, by using $LU{T}_M,$ only for the positions $LU{T}_M(\ell ,k)=0$, we can employ Equation (12) to generate the mask. Let us denote $m_{\kappa }^{\ell }$ the number of positions for which Equation (12) is employed. Since $m_{\kappa }^{\ell }$ depends on the structure of the found unique combination, a variable-length representation would be required to store in memory the LUT-based structure of ML-LUT efficiently. However, to guarantee a fixed-length representation, we propose to store each updated mask using a fixed-length representation of $m^{\ell }=min(\ell N_{uc}^{\ell },N_t)$ bits. Hence, Equation (13) is updated as Equation (17), and Equation (14) is updated as Equation (18).

$${\mathcal{L}}_{LU{T}_{\ell }}^{ML}=(m_{\ell }+8\ell )N_{uc}^{\ell }$$

(17)

$${\mathcal{L}}^{ML}={\mathcal{L}}_M^{2L}+{\mathcal{L}}_{Mask}^{ML}+\sum _{\ell =1}^{N_t}{\mathcal{L}}_{LU{T}_{\ell }}^{ML}=N_g(n_{\ell }+n_{\kappa }^M)+N_t\pi +\sum _{\ell =1}^{N_t}(m^{\ell }+8\ell )N_{uc}^{\ell }$$

(18)

3.2.4. Variable-Length Solution

In this work, we also studied the possibility of relaxing the constraint of fixed-length representation for encoding the LUT-based structure and introducing a variable-length multi-level LUT solution called MLv-LUT, see Figure 9. MLv-LUT simply modifies the fixed-length representation version, ML-LUT, by introducing two variable-length strategies for storing each $LU{T}_{\ell }$ without where the zero-padding procedure is removed from the generation of the filtered mask and the last non-zero symbol $t_{N_t-1}.$

Firstly, instead of storing each filtered mask using a fixed number of bits, $m_{\kappa },$ we propose to store each filtered mask using the exact number of bits, $m_{\kappa }^{\ell },$ and avoid the zero-padding process. The cost of storing the filtered mask of $LU{T}_{\ell }$ is computed by (19).

$${\mathcal{L}}_{LU{T}_{\ell }}^{MLv}=\sum _{\kappa =1}^{N_{uc}^{\ell }}\left(m_{\kappa }^{\ell }+8\ell N_{uc}^{\ell }\right)$$

(19)

Secondly, the last non-zero symbol $t_{N_t-1}$ can be efficiently represented using less then 8 bits by removing the zero-padding process of $V_{i_g{j}_g}^v.$ In such case, the last group of ternary symbols in $V_{i_g{j}_g}^v$ is $[s_{5N_t-5}\dots s_N]$ and has a length of $r_l$ symbols, where $r_l$ is computed by Equation (20). We can now update Equation (3) as Equation (21) for remapping $t_{N_t-1}.$ If $r_l<5,$ then $t_{N_t-1}\in \{0,1,\dots ,3^{r_l}-1\}$ and, instead of 8 bits, only $n_t$ bits are needed to store $t_{N_t-1},$ where $n_t$ is computed using Equation (22). Let us denote $N_{nzl}^{\ell }$, the number of unique combinations in $LU{T}_{\ell }$ which contain a non-zero symbol on the last position. The gain obtained by employing such a strategy is computed by (23). Due to the variable-length representation of the LUT-based structure, an additional cost of ${\mathcal{L}}_{vl}^{MLv}$ bits, computed by (24), is required to signal which LUTs are included in the LUT set, i.e., for which $N_{uc}^{\ell }>0.$

$$r_l=N-5(N_t-1)$$

(20)

$$t_{N_t-1}=\sum _{r=0}^{r_l}{3}^{r_l-r}{s}_{5k+r}$$

(21)

$$n_t=\lceil {log}_2{3}^{r_l}\rceil$$

(22)

$${\mathcal{L}}_{gain}^{MLv}=(8-n_t)N_{nzl}^{\ell }$$

(23)

$${\mathcal{L}}_{vl}^{MLv}=N_t$$

(24)

Finally, MLv-LUT stores in memory $\mathbf{I}$ using ${\mathcal{L}}^{MLv}$ bits, where ${\mathcal{L}}^{MLv}$ is computed by (25) using (11), (16), (19), (23), and (24).

$$\begin{array}{cc}\hfill {\mathcal{L}}^{MLv}\hfill & ={\mathcal{L}}_M^{2L}+{\mathcal{L}}_{Mask}^{ML}+{\mathcal{L}}_{vl}^{MLv}+\sum _{\ell =1}^{N_t}\left({\mathcal{L}}_{LU{T}_{\ell }}^{MLv}-{\mathcal{L}}_{gain}^{MLv}\right)\hfill \\ \hfill & \hfill =N_g(n_{\ell }+n_{\kappa }^M)+N_t\pi +N_t+\sum _{\ell =1}^{N_t}\left(\sum _{\kappa =1}^{N_{uc}^{\ell }}{m}_{\kappa }^{\ell }+8\ell N_{uc}^{\ell }-(8-n_t)N_{nzl}^{\ell }\right)\hfill \end{array}$$

(25)

3.3. Algorithmic Details

The encoder implementation of the proposed 2L-LUT/ML-LUT fixed-length representation is presented in Algorithm 1. The threshold $\mathcal{T}$ is used to select the appropriate solution based on the group size information. One can note that the index matrix is initially encoded as bitstream ${\mathcal{B}}_{\mathcal{M}}$ using steps 5–22 in Algorithm 1. The proposed LUT-based representation is encoded as bitstream ${\mathcal{B}}_{\mathcal{LUT}}$ using steps 23–39 in Algorithm 1. Bitstream ${\mathcal{B}}_{\mathcal{I}}$ is finally updated using steps 40–41 in Algorithm 1 by removing the unnecessary bits based on the information provided by the proposed LUT-based representation. The output bitstream, $\mathcal{B},$ concatenates ${\mathcal{B}}_{\mathcal{LUT}}$ and ${\mathcal{B}}_{\mathcal{M}}$ in this order.

Algorithm 1:  Encode using ML-LUT

The pseudo-code provided by Algorithm 1 is used to encode the input image and store the data on the disk, while (14) and (18) compute the total memory needed to store in memory all the necessary information.

The decoder implementation of the proposed 2L-LUT/ML-LUT fixed-length representation is presented in Algorithm 2. Similarly, the threshold $\mathcal{T}$ is used to select the appropriate solution based on the group size information. The proposed LUT-based representation is first decoded from the first part of $\mathcal{B}$ using steps 7–24 in Algorithm 2. The index matrix, $\mathbf{M},$ is decoded from the remaining part of $\mathcal{B}$ using steps 26–29 in Algorithm 2, i.e., ℓ and ${\kappa }_{\ell }$ are decode on steps 28 and 29 in Algorithm 2, respectively. Note that the inverse remap process consists in first representing each symbol $t_k$ in $V_{i_g{j}_g}$ in base 3 to generate five ternary symbols and then selecting $G_{i_g{j}_g}$ with the first $N_t$ ternary symbols. Finally, $\mathbf{I}$ is generated using the proposed LUT-based representation and $\mathbf{M}$ using steps 30–36 in Algorithm 2.

Algorithm 1 and Algorithm 2 are used to introduce the encoder and decoder of the proposed fixed-length ML-LUT method, respectively. In the case of the proposed variable-length MLv-LUT method, the two algorithms may be further modified using the algorithmic description presented in Section 3.2.4, i.e., by modifying steps 33–34 in Algorithm 1 and steps 21–22 in Algorithm 2.

The size of $\mathcal{B},$ i.e., the compressed file size, is usually smaller than the total memory usage size computed by ${\mathcal{L}}^{2L}$ or ${\mathcal{L}}^{ML}.$ In this work, ${\mathcal{L}}^{1L},{\mathcal{L}}^{2L},$ and ${\mathcal{L}}^{ML}$ are computed to report the total memory usage of each proposed fixed-length method, while ${\mathcal{L}}^{MLv}$ is computed to report the optimal storage size of the proposed framework.

Algorithm 2:  Decode using ML-LUT

4. Experimental Evaluation

4.1. Experimental Setup

In our work, the experimental evaluation is carried out on the large-scale outdoor stereo event camera datasets [21], called DSEC, containing a set of 82 asynchronous event sequences captured for network training using the Prophesee Gen3.1 event sensor placed on top of a moving car, having a $W\times H=640\times 480$ resolution.

In this work, the total memory usage of the three proposed fixed-length representations, 1L-LUT, 2L-LUT, and ML-LUT, and the compressed file size of the proposed variable-length representation, MLv-LUT, are reported. All proposed methods are implemented in the MATLAB programming language. One can note that both 2L-LUT and ML-LUT results are reported as ML-LUT, where the corresponding method is selected based on the group size, $N.$ An EF sequence is generated using the sum-accumulation process for each asynchronous event sequence and each time-window size $\Delta$. Four frame rates are studied for generating the EFs: (i) $\Delta =5555$ $\mathsf{\mu }$s (180 fps); (ii) $\Delta =1000$ $\mathsf{\mu }$s (${10}^3$ fps); (iii) $\Delta =100$ $\mathsf{\mu }$s (${10}^4$ fps); and (iv) $\Delta =1$ $\mathsf{\mu }$s (${10}^6$ fps), i.e., all acquired asynchronous events are collected by EFs as all events having the same timestamp are collected by one EF. The proposed methods are designed to store-on-disk/represent-in-memory each EF separately. The performance of the proposed framework is studied using a large variety of group sizes. The group configuration provides RA to blocks of pixels having the $w\times h$ block size set as follows: (g${}_1$) $8\times 4$; (g${}_2$) $16\times 4$; (g${}_3$) $8\times 8$; (g${}_4$) $16\times 8$, (g${}_5$) $64\times 4$, (g${}_6$) $16\times 16$, (g${}_7$) $32\times 32$, and (g${}_8$) $64\times 32$. More exactly, ML-LUT selects the 2L-LUT method for (g${}_1$)–(g${}_6$) and the ML-LUT method for (g${}_7$) and (g${}_8$).

The performance of the proposed lossless coding framework is compared with the following state-of-the-art methods, which encode lossless intensity-like event representation that remaps a sequence of five EFs (see [18]):

(1) 

The HEVC standard [17] using the FFmpeg implementation [39] which was employed using the following command:

$$\begin{array}{cc}\hfill \mathit{ffmpeg}& \hfill -\mathit{f}\mathit{rawvideo}-\mathit{vcodec}\mathit{rawvideo}-\mathit{s}\mathit{W}\mathit{x}\mathit{H}-\mathit{r}\mathit{30}-\mathit{pix}\_\mathit{fmt}\mathit{gray}-\mathit{i}\mathit{in}.\mathit{yuv}\hfill \\ \hfill & -\mathit{c}:\mathit{v}\mathit{libx}\mathit{265}-\mathit{x}\mathit{265}-\mathit{params}\mathit{lossless}=\mathit{1}\mathit{out}.\mathit{mp}\mathit{4}\hfill \end{array}$$

(26)

(2) 

The Versatile Video Codec (VVC) standard [36] using the VVC Test Model [40] implementation which was employed using the following command:

$$\begin{array}{cc}\hfill \mathit{EncoderAPP}& -\mathit{c}\mathit{encoder}\_\mathit{intra}\_\mathit{vtm}.\mathit{cfg}-\mathit{c}\mathit{lossless}.\mathit{cfg}-\mathit{FrameRate}=\mathit{30}-\mathit{wdt}\mathit{W}-\mathit{hgt}\mathit{H}\hfill \\ \hfill & -\mathit{InputChromaFormat}=\mathit{400}-\mathit{FramesToBeEncoded}=\mathit{1}-\mathit{InputBitDepth}=\mathit{8}\hfill \\ \hfill & -\mathit{OutputBitDepth}=\mathit{8}-\mathit{BDPCM}=\mathit{1}-\mathit{i}\mathit{in}.\mathit{yuv}-\mathit{b}\mathit{out}.\mathit{vvc}-\mathit{o}\mathit{out}.\mathit{yuv}\hfill \end{array}$$

(27)

(3) 

The Context-based, Adaptive, Lossless Image Codec (CALIC) lossless image codec [37];

(4) 

The Free Lossless Image Format (FLIF) lossless image codec [38]; and with our performance-oriented prior work Lossless Compression of Event Camera Frames (LCECF) [18], where eight EFs are represented as a pair of an event map image (EMI) and a concatenated polarity vector.

The Raw data size is computed using 2 bits per pixel, i.e., each pixel symbol is stored using 2 bits; therefore, the size of one EF is $2HW$ bits. Moreover, a similar procedure as in [18,19] is employed: the EF sequences generated for only the first $20$s, $2$s, and $20$ms are extracted from each asynchronous event sequence for $\Delta =1000$ $\mathsf{\mu }$s, $\Delta =100$ $\mathsf{\mu }$s, and $\Delta =1$ $\mathsf{\mu }$s, respectively, i.e., a maxim of 20,000 EFs are generated for each sequence.

The coding performance of all codecs is measured using the compression ratio ($CR$) metric defined as follows:

$$CR=\frac{\mathrm{Raw}\mathrm{data}size}{\mathrm{Compressed}\mathrm{file}size}.$$

(28)

For ML-LUT, Equation (28) is computed using the size of output bitstream, $\mathcal{B},$ generated by Algorithm 1. See Section 4.2 for the lossless compression results over DSEC.

The memory usage performance of the proposed fixed-length codecs is measured using the compression ratio—memory usage ($C{R}_{MU}$) metric defined as follows:

$$C{R}_{MU}=\frac{\mathrm{Raw}\mathrm{data}size}{\mathrm{Total}\mathrm{memory}\mathrm{usage}size}.$$

(29)

More exactly, Equation (29) is computed using (8), (14), (18), and (25), for the proposed fixed-length representation solutions, 1L-LUT, 2L-LUT, ML-LUT, and MLv-LUT, respectively. See Section 4.3 for the memory usage results over DSEC for the proposed fixed-length representations. See Section 4.4 for the average runtime results per EF over DSEC.

4.2. Lossless Compression Results

Figure 10 shows the lossless compression results over DSEC for four different values of the time window of the spatiotemporal neighborhood. One can note that on one hand, when $\Delta$ is very small ($\Delta =1$ $\mathsf{\mu }$s), see Figure 10a, a larger block size provides improved performance as the EF is sparsely populated with event frames as a small amount of information is stored in each EF, while a small block size provides a limited improvident as a few bits are still necessary to signal empty group (i.e., full of non-event symbols). While on the other hand, when $\Delta$ is much larger ($\Delta =5555$ $\mathsf{\mu }$s), see Figure 10d, the largest block size might not provide the best performance as the EF size is limited to $W\times H,$ and the cost of encoding ℓ might become too high as too many LUTs are generated with only a few unique combinations. Figure 10 shows that, in general, the $32\times 32$ group size provides the best results for any of the selected time-window sizes. Therefore, the ML-LUT $32\times 32$ solution was selected as the anchor fixed-length representation, and the DSEC sequences are sorted in ascending order of the ML-LUT $32\times 32$ result.

Table 1. Lossless compression results over DSEC for ML-LUT using different group sizes.

Method		ML-LUT 8 × 4	ML-LUT 16 × 4	ML-LUT 8 × 8	ML-LUT 16 × 8	ML-LUT 64 × 4	ML-LUT 16 × 16	ML-LUT $32\times 32$	ML-LUT 64 × 32
$\Delta =1$ $\mathsf{\mu }$s	CR	15.04	29.64	29.67	58.32	113.33	111.17	310.13	312.98
	Improv.	−95.15%	−90.44%	−90.43%	−81.20%	−63.46%	−64.15%	–	0.92%
$\Delta =100$ $\mathsf{\mu }$s	CR	5.97	9.92	9.89	13.92	15.83	16.07	27.18	27.67
	Improv.	−78.03%	−63.52%	−63.61%	−48.80%	−41.76%	−40.88%	–	1.79%
$\Delta =1000$ $\mathsf{\mu }$s	CR	3.30	4.45	4.46	5.22	5.47	5.58	6.84	6.43
	Improv.	−51.76%	−34.96%	−34.84%	−23.59%	−20.04%	−18.37%	–	−5.95%
$\Delta =5555$ $\mathsf{\mu }$s	CR	2.17	2.62	2.60	2.88	3.03	3.03	3.15	3.00
	Improv.	−31.12%	−16.69%	−17.39%	−8.36%	−3.81%	−3.62%	–	−4.51%

shows the average compression results over DSEC, where the bold font marks the best results. One can note that ML-LUT $32\times 32$ is able to provide a CR performance of at least $3.15$ at $\Delta =5555$ $\mathsf{\mu }$s and up to $310.13$ at $\Delta =1$ $\mathsf{\mu }$s. If the chip memory requirements are stricter, then a small group size can be used, however, the CR performance decreases by up to $95.15\%$ at $\Delta =1$ $\mathsf{\mu }$s. One can note that the proposed method is able to provide a minimum CR performance of at least 2, i.e., the ML-LUT can offer the sensor to store in memory at least two compressed EF and apply other pre-processing methods (using a block-processing approach) compared with a single EF using the raw representation. The results also show that a larger group size might not provide improved performance as a large group size and the possible number of unique combinations increases too much.

Figure 11 shows the lossless compression results over DSEC for comparison with state-of-the-art methods, using different values of $\Delta .$ One can note that when $\Delta$ is very small ($\Delta =1$ $\mathsf{\mu }$s), see Figure 11a, the performance of the proposed ML-LUT $32\times 32$ method is worse than the state-of-the-art methods as the fixed-length representation constraint is must be satisfied by still using a few bits to signal empty groups. This small extra cost per group adds up to a high extra cost compared with state-of-the-art methods; however, ML-LUT $32\times 32$ provides an average CR compared with the raw data representation. While on the other hand, when $\Delta$ is much larger, see Figure 11d, the proposed fixed-length representation ML-LUT $32\times 32$ provides competitive results compared with the state-of-the-art variable-length representations, especially to the video coding standards. One can note that ML-LUT must work under powerful constraints: must provide a fixed-length representation, i.e., for each group of pixels, we must allocate an equal number of bits, although maybe some groups of pixels can be encoded using a much less bitrate; must have a low-complexity so it is suitable for integration into VLP event-processing chips, i.e., it can be written on hardware (e.g., the use of basic coding concepts such as arithmetic coding or adaptive Markov modeling are too complex); and must provide an efficient memory representation, i.e., the coding gains must be large enough to justify the introduction of the proposed coding method in the sensor.

Table 2. Lossless compression results over DSEC for comparison with state-of-the-art.

Method		HEVC [17]	VVC [36]	CALIC [37]	FLIF [38]	LCECF [18]	ML-LUT $32\times 32$
Representation		Variable-Length	Variable-Length	Variable-Length	Variable-Length	Variable-Length	Fixed-Length
Chip Integration		SoC	SoC	SoC	SoC	SoC	VLP/ESP
$\Delta =1$ $\mathsf{\mu }$s	CR	1715.40	1507.25	2309.98	2775.60	5533.72	310.13
	Improv.	453.12%	386.01%	644.84%	794.98%	1684.32%	–
$\Delta =100$ $\mathsf{\mu }$s	CR	46.21	50.17	45.12	67.52	80.10	27.18
	Improv.	70.01%	84.58%	66.00%	148.42%	194.70%	–
$\Delta =1000$ $\mathsf{\mu }$s	CR	7.26	7.84	7.21	10.25	12.87	6.84
	Improv.	6.14%	14.62%	5.41%	49.85%	88.16%	–
$\Delta =5555$ $\mathsf{\mu }$s	CR	3.24	3.49	3.51	4.40	5.15	3.15
	Improv.	2.86%	10.79%	11.43%	39.68%	63.49%	–

shows the average compression results over DSEC for comparison with state-of-the-art, where the bold font marks the best results. One can note that the proposed ML-LUT $32\times 32$ solution, although offering a fixed-length representation, is able to provide a close CR performance compared with variable-length video coding standards and variable-length state-of-the-art image codecs for $\Delta =1000$ $\mathsf{\mu }$s and $\Delta =5555$ $\mathsf{\mu }$s, i.e., at frequencies below 1 KHz. When $\Delta$ is too small, the fixed-length representation feature of the proposed framework affects the CR performance.

4.3. Memory Usage Results

Figure 12 shows the memory usage results over DSEC for 1L-LUT using different group sizes. One can note that for each $\Delta$, a different group size provides the best performance. A large group size is affecting the overall performance as too much information is stored by the LUT-based structure. However, the proposed 1L-LUT solution has the lowest complexity, and the provided performance might be enough for some types of applications.

Figure 13 shows the memory usage results for ML-LUT and MLv-LUT using different group sizes. Note that the $32\times 32$ group size usually provides the best performance. For $\Delta =1$ $\mathsf{\mu }$s, a larger group size provides a much better performance than a smaller group size. Moreover, variable-length representation solutions provide only a small improvement compared with the corresponding fixed-length representation solutions.

Table 3. Memory usage results over DSEC for ML-LUT and MLv-LUT using different group sizes.

Method		ML-LUT 8 × 4	ML-LUT 16 × 4	ML-LUT 8 × 8	ML-LUT 16 × 8	ML-LUT 64 × 4	ML-LUT 16 × 16	ML-LUT 32 × 32	MLv-LUT 32 × 32	ML-LUT 64 × 32	MLv-LUT 64 × 32
$\Delta =1 \mathsf{\mu }$s	$C{R}_{MU}$	15.04	29.66	29.68	58.38	113.58	111.42	353.70	353.82	411.43	411.49
	Improv.	−95.75%	−91.61%	−91.61%	−83.49%	−67.89%	−68.50%	–	0.03%	16.32%	16.34%
$\Delta =100 \mathsf{\mu }$s	$C{R}_{MU}$	5.97	9.91	9.89	13.91	15.83	16.07	26.38	27.77	28.35	28.58
	Improv.	−77.37%	−62.43%	−62.51%	−47.27%	−39.99%	−39.08%	–	5.27%	7.47%	8.34%
$\Delta =1000 \mathsf{\mu }$s	$C{R}_{MU}$	3.27	4.44	4.44	5.20	5.46	5.57	6.56	6.91	6.44	6.50
	Improv.	−50.15%	−32.32%	−32.32%	−20.73%	−16.77%	−15.09%	–	5.34%	−1.83%	−0.91%
$\Delta =5555 \mathsf{\mu }$s	$C{R}_{MU}$	2.12	2.61	2.59	2.86	3.01	3.02	2.92	3.17	2.96	3.02
	Improv.	−27.40%	−10.62%	−11.30%	−2.05%	3.08%	3.42%	–	8.56%	1.37%	3.42%

shows the average compression results for ML-LUT and MLv-LUT, where the bold font marks the best results. Note that the memory usage results are similar to the lossless compression results; see Table 1. The variable-length representation solutions always provide improved performance compared with the corresponding fixed-length representation solutions.

4.4. Runtime Results

Figure 14 shows the encoding runtime results.

Table 4. Encoding runtime results over DSEC for ML-LUT using different group sizes.

Method		ML−LUT 8 × 4	ML−LUT 16 × 4	ML−LUT 8 × 8	ML−LUT 16 × 8	ML−LUT 64 × 4	ML−LUT 16 × 16	ML−LUT 32 × 32	ML−LUT 64 × 32
$\Delta =1$ $\mathsf{\mu }$s	time (ms/EF)	48.44	31.19	30.70	26.48	23.52	22.76	19.81	19.68
	Improv.	144.52%	57.45%	54.97%	33.67%	18.73%	14.89%	–	−0.66%
$\Delta =100$ $\mathsf{\mu }$s	time (ms/EF)	132.87	99.05	97.24	83.54	79.14	75.21	60.57	56.05
	Improv.	119.37%	63.53%	60.54%	37.92%	30.66%	24.17%	–	−7.46%
$\Delta =1000$ $\mathsf{\mu }$s	time (ms/EF)	217.68	154.18	150.69	115.08	92.41	88.72	69.82	65.18
	Improv.	211.77%	120.82%	115.83%	64.82%	32.35%	27.07%	–	−6.65%
$\Delta =5555$ $\mathsf{\mu }$s	time (ms/EF)	200.99	100.12	98.08	62.45	46.67	45.09	35.51	32.49
	Improv.	466.01%	181.95%	176.20%	75.87%	31.43%	26.98%	–	−8.50%

shows the average compression results for ML-LUT, where a larger group size always provides a smaller runtime. The hardware implementation of the proposed method can provide a much smaller runtime as the proposed framework is designed to employ only low-complexity coding techniques and a reduced number of operations, compared with state-of-the-art methods which are developed to compute more complex structures and employ complex coding techniques. To be able to hardware write such complex methods, several other optimization techniques must be applied while accepting a drop in the coding performance and paying a high price for developing such a specialized chip. Therefore, here, these complex methods are evaluated as unsuitable for integration into VLP chips.

5. Conclusions

The paper proposed a novel low-complexity lossless compression framework for encoding synchronous EFs by introducing a novel memory-efficient fixed-length representation suitable for hardware implementation in the VLP event-processing chip. The proposed framework first partitions the ternary EFs into groups and remaps the symbols. Several solutions are proposed using different levels of LUTs. 2L-LUT is the low-complexity lossless compression solution for encoding EFs, which provides fixed-length representation based on two-level LUTs. ML-LUT provides an improved fixed-length representation based on multi-level LUTs using very-large groups of pixels. The proposed framework provides RA to any group of pixels using a fixed-length representation.

The experimental evaluation demonstrates that the proposed fixed-length framework provides at least two times the compression ratio relative to the raw EF representation and a close performance compared with the traditional variable-length lossless compression methods, such as CALIC and FLIF, and with the variable-length video coding standards, HEVC and VVC, for lossless compression of the ternary EFs generated at frequencies below 1 KHz. To our knowledge, the paper is the first to explore low-complexity lossless compression solutions which provide fixed-length representation of synchronous EFs, suitable for integration into very-low-power processing chips.