Flexible Quantization for Efficient Convolutional Neural Networks

Abstract

This work focuses on the efficient quantization of convolutional neural networks (CNNs). Specifically, we introduce a method called non-uniform uniform quantization (NUUQ), a novel quantization methodology that combines the benefits of non-uniform quantization, such as high compression levels, with the advantages of uniform quantization, which enables an efficient implementation in fixed-point hardware. NUUQ is based on decoupling the quantization levels from the number of bits. This decoupling allows for a trade-off between the spatial and temporal complexity of the implementation, which can be leveraged to further reduce the spatial complexity of the CNN, without a significant performance loss. Additionally, we explore different quantization configurations and address typical use cases. The NUUQ algorithm demonstrates the capability to achieve compression levels equivalent to 2 bits without an accuracy loss and even levels equivalent to ∼1.58 bits, but with a loss in performance of only ∼0.6%.

1. Introduction

The computational complexity of convolutional neural networks (CNNs) has been increasing since their inception, mainly due to their usage in solving more challenging problems, which, in turn, leads to the incorporation of a higher number of parameters. Furthermore, applications now have more stringent requirements, and the platforms where CNNs are deployed face tighter constraints in terms of resource budgets. These constraints include factors such as the size and bandwidth (BW) of external memory, on-chip memory capacity, the number of processing elements (PEs), and maximum power consumption. When considering edge devices, which are gaining popularity, these constraints become even more restrictive. In general, deploying a CNN on one of these embedded platforms is a challenging task, if not impossible. However, this issue can be addressed by applying a technique known as network compression, which allows the CNN to fit within the platform’s constraints. One possible classification for compression techniques, starting from a high level and going down to a lower level, is as follows:

• Architecture modifications: The goal of this compression technique is to define an efficient high-level architecture for the NN. Some examples of architectural modifications include Inception [1] and residual blocks [2], which propose novel nonlinear networks as alternatives to linear networks like VGG [3]. Additionally, MobileNets [4] utilize depthwise convolutions to enhance computational efficiency, while EfficientNets [5] offer a method to uniformly scale the depth, width, and resolution of a network. Another method for optimizing the architecture is network architecture search (NAS) [6], which falls under the subfield of Auto-ML. NAS automates the search process. Knowledge distillation (KD) [7] is another method that contributes to NN compression. It enables the transfer of knowledge from a larger model to a smaller one, resulting in a more compact network with an improved architecture tailored to address the specific problem effectively. Co-design is another technique that falls within this category [8].
• Pruning: This technique is based on the idea of removing unimportant parts of the NN [9,10]. Typically, it is applied after training, when it becomes possible to determine what is important and what is not. Pruning can be performed in various ways, including layer-wise, filter-wise, shape-wise, channel-wise, and element-wise [10]. It can be classified into two types: structured and unstructured pruning [10].
• Quantization: This approach aims to find an efficient way of representing or storing the weights and activations [8,11,12,13,14,15,16,17,18,19,20,21,22,23,24]. Popular quantization techniques include uniform quantization (UQ), non-uniform quantization (NUQ) or weight-sharing, and matrix/tensor decomposition. The KD process may also be applied to optimize the quantization parameters.

Most of these methods can be used together to jointly compress the NN.

The main objective of this work is to address the quantization challenges of CNNs in order to improve CNN efficiency for embedded systems and edge devices; this leads us to 1. develop a flexible quantization approach that balances non-uniform and uniform quantization benefits, aiming for high compression without losing accuracy; and 2. state and solve the optimization problem that lets us find the best quantization parameters for a CNN.

To achieve the objective, it is proposed to apply a hardware-aware quantization. In this work, this means that the resulting compressed network satisfies two key conditions: 1. It can be efficiently implemented using fixed-point (FxP) hardware, and 2. it helps reduce the hardware resource needs, such that the constraints are met. More specifically, our contributions are as follows:

• Non-uniform uniform quantization (NUUQ): The main idea behind NUUQ is to decouple the number of quantization levels from the number of quantization bits, combined with a clustered based quantization and efficient data clipping. Consequently, there is an extra optimization parameter: the number of quantization levels (or clusters). A mixed-precision (MP) approach is also considered in NUUQ, but with a significant advantage, which is that the granularity of the MP quantization may be different for quantization bits and quantization levels. Moreover, the impact in the hardware complexity when increasing the quantization levels granularity is very little. So it is possible to increase it, but with little impact in the hardware complexity. This results in more quantization flexibility compared to the classical quantization. The overall result is a compressed network that can be efficiently implemented using FxP hardware.
• Mid-level metrics: As a consequence of decoupling the number of quantization bits and levels, new more representative metrics are needed. We introduce three mid-level complexity metrics that are more hardware-focused than the usual metrics, yet remain implementation-independent. These metrics are utilized to constrain our optimization problem, and they effectively relate the platform constraints to the application requirements.
• Exploration of quantization parameters and practical applications: We conduct an analysis of how various quantization parameters influence CNN performance, providing insights for their optimal selection. Additionally, we present real-world use cases, showcasing our method’s practical benefits in deploying efficient CNNs

When implementing quantization in CNN, there are numerous possibilities to consider. For instance, the choice between UQ or NUQ for each data source, such as kernels, biases, and activations, introduces various possibilities. Additionally, granularity can be adjusted both in terms of the number of bits and quantization levels. It is also possible to combine different types of quantization schemes and granularities for each layer, further expanding the options. In this work, we aim to explore and compare several of these configurations, selecting those that we deem most representative and suitable for real life applications These configurations are detailed in

Table 1. Summary of the quantization configurations explored in this study. Some cases might involve fine-tuning (FT). The abbreviated names for each case are displayed in the last column.

Description	Arithmetic	FT	Name
Original model (baseline)	FlP	No	OM
UQ with layer-wise MP [12,16,17,20,25,26,27,28,29,30]	FxP	No	UQ-L
	FxP	Yes	UQ-L-FT
NUUQ with layer-wise MP [this work]	FxP	No	NUUQ-LL
	FxP	Yes	NUUQ-LL-FT
NUUQ with channel-wise level selection and layer-wise bit selection [this work]	FxP	No	NUUQ-LC
	FxP	Yes	NUUQ-LC-FT

.

This work is structured as follows: Firstly, we present related work in Section 2. Secondly, our proposal is developed in Section 3. In Section 4, we present the results, and finally, we draw conclusions in Section 5.

2. Related Work

Neural networks are implemented on various platforms today, including classic microprocessors with or without machine learning (ML) support, GPUs (graphics processing Units), GPGPUs (general-purpose graphics processing Units), FPGAs (field-programmable gate arrays), ML-specific FPGAs, and ASICs (application-specific integrated circuits). Each of these platforms is suitable for different applications or domains. Additionally, each has distinct advantages, disadvantages, requirements, and limitations [8,31,32].

In terms of the architectures employed, we can highlight that the use of either unrolled or iterative architectures is possible. The iterative approach offers greater flexibility but tends to be less hardware optimized. On the other hand, the unrolled approach is more specific, providing optimizations that result in less flexibility. Moreover, for scenarios where low latency is needed, typically with a batch size of 1, the unrolled method is advantageous compared to its iterative counterpart [33].

In terms of the implementation, there are two major approaches: the use of floating-point (FlP) or FxP arithmetic. FlP arithmetic provides greater flexibility, while FxP, being optimized for a specific application, achieves higher efficiency levels in terms of energy consumption, hardware size and other resource utilization. The use of different FxP arithmetic for different parts of the CNN is proposed in order to improve the network performance. This approach is known as MP [25,26,27,34,35]. And this is the main reason why unrolled architectures are better using hardware resources. The MP approach has also reached the iterative architectures. The work [36] serves as an example, where an iterative, FxP, ASIC architecture is proposed, seeking efficient resource utilization. The FxP arithmetic, is limited to 2, 4, 8, or 16 bits, and the number of PEs is 8, 4, 2, or 1, respectively.

Works like [32,37,38,39,40,41,42,43,44,45] propose to use the weight sharing (WS) approach to increase the weight reusability. This enables the storage of a reduced set of weights along with a set of indices, rather than storing all the individual weights separately. As the indices representation requires fewer bits than the weights, it results in a reduction in memory footprint. To implement WS, clustering algorithms are usually used, with the most popular one being the K-means clustering. The WS approach is commonly employed in conjunction with FlP arithmetic. As the probability of each weight is not even along all of them, coding results are useful for further reducing the memory footprint.

The most widely used coding scheme is Huffman coding (HC). This has been used in works like [46,47,48,49].

The utilization of FT in order to recover accuracy loss has become essential whenever feasible; this is when training data are available [50,51,52]. The FT technique has also been proposed in [16] to recover performance after applying quantization, which is especially relevant for this work.

Quantization algorithms can primarily be classified into UQ and NUQ. UQ algorithms hold a significant advantage in being easily implementable using FxP arithmetic, which reduces hardware complexity, thereby enhancing performance and lowering latency and power consumption [30,36,53]. On the other hand, NUQ algorithms typically utilize hardware with FlP arithmetic, which has greater complexity compared to FxP hardware, but they offer the advantage of achieving higher levels of accuracy [13,37].

Nowadays, state-of-the-art hardware for large-scale deployment of ML algorithms for inference is primarily dominated by FxP hardware, spanning across CPUs, GPUs, TPUs, FPGAs, and ASICs [8,31,32,33,53]. This puts NUQ algorithms at a disadvantage. To overcome this challenge, Ref. [18] proposes a combination of UQ and NUQ. Similarly, this work proposes NUUQ, which also combines both approaches; however, in contrast to [18], this study is not confined to $2^n$ quantization levels, nor does it enforce equidistant levels, which resembles NUQ more closely.

When using UQ, it is very important to properly choose the quantization precision, specially for the lower-value regions [9,15,54,55,56,57,58]. This process depends on several variables: the scale factor, the range of values, and the number of allocated bits. In [15], it is also proposed to clip the higher values in order to improve the lower-value resolution. Moreover, this clipping parameter, $\alpha$-value, is optimized during training.

3. Proposed Quantization Method and Metrics

3.1. Quantization Method

The NUUQ approach proposes the combination of NUQ and UQ techniques to achieve non-uniform quantization while remaining efficient for implementation using FxP representation. This is accomplished through a layer-wise approach involving clipping, NUQ (utilizing K-means, although other methods are also viable), followed by subsequent UQ applied to the cluster centers. Additionally, the cluster centers are encoded using HC to improve memory footprint efficiency. Finally, FT is optionally applied to recover any performance lost during the quantization process. Figure 1 provides an example of how the NUUQ quantization function ($Q^{\mathrm{NUU}}$) is formed, and Figure 2 demonstrates the outcome of applying NUUQ to the weights of a specific layer.

It is important to note that both the allocation of bits and number of clusters is performed on a layer-wise basis. This simplifies hardware implementation since PEs and decoders (in the case of hierarchical coding usage) can be utilized across all channels. While bit and number of cluster assignment is conducted on a layer-wise basis, clustering itself can be executed either on a layer-wise or channel-wise basis, with the expectation that the latter approach may yield improved results in terms of performance and spatial complexity.

3.2. Mid-Level Metrics and Optimization Problem

During the implementation process, it is important to have notion of how our design variables relate to our constraints. For example, if we assume that the weights of the CNN are stored in external memory, the number of bits used for weights quantization are directly related to the BW needs. In this work, the number of quantization bits and quantization levels have no longer a tight relationship, so it is important to see that the usual metrics are no longer representative. For example, the spatial complexity of the weights, now depends has a tight dependence with the number of clusters, while the size (in bits) of the activations depends on the number of bits. For this reason, in this work, we introduce three mid-level complexity metrics that are more hardware-focused and yet remain implementation-independent.

• Binary operations complexity (BOC): This metric represents the binary time complexity (hardware complexity) of the layer, measured in binary operations. Is represented by $c^{\mathrm{T}}$.
• Binary weight complexity (BWC): This metric quantifies the spatial complexity of the weights (K and B), measured in bits. Is represented by $c^{\mathrm{W}}$.
• Binary activation complexity (BAC): This metric measures the spatial complexity of the layer activation, expressed in bits. Is represented by $c^{\mathrm{A}}$.

These metrics are derived for our approach as follows. First, it is assumed that the output of each layer can be efficiently computed using multiply and accumulate (MAC) operations. In this situation, we are interested in knowing the sizes of I, K, B, and A ($n^{\mathrm{I}}$, $n^{\mathrm{K}}$, $n^{\mathrm{B}}$ y $n^{\mathrm{A}}$), as well as the total number of MACs ($n^{\mathrm{M}}$) for each layer. In [59], these values are derived for the Conv2D and dense layers. We will also assume that PEs implement MAC operations using FxP arithmetic according to Figure 3, which is very representative for both ASIC and FPGA).

Next, it is common to consider that the binary complexity for a sum of two words with m and n bits, respectively, is equal to $max(m,n)$, and the binary complexity for the multiplication of those two words is equal to $\left(mn\right)$. Finally, we are in a position to derive the formulas for the proposed metrics:

$$\begin{array}{cc}\hfill c^{\mathrm{T}}& =n^{\mathrm{M}}{b}^{\mathrm{K}}{b}^{\mathrm{I}}+n^{\mathrm{M}}{b}^{\mathrm{X}}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill c^{\mathrm{W}}& =n^{\mathrm{K}}{L}^{\mathrm{K}}+k^{\mathrm{K}}{b}^{\mathrm{K}}+n^{\mathrm{B}}{L}^{\mathrm{B}}+k^{\mathrm{B}}{b}^{\mathrm{B}}\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill c^{\mathrm{A}}& =n^{\mathrm{A}}{b}^{\mathrm{A}}\hfill \end{array}$$

(3)

According to our NUUQ quantization approach, in each layer, we have these variables: clipping values (${\alpha }^{\mathrm{K}}$, ${\alpha }^{\mathrm{B}}$, ${\alpha }^{\mathrm{A}}$), number of bits ($b^{\mathrm{K}}$, $b^{\mathrm{B}}$, $b^{\mathrm{A}}$), and number of clusters ($k^{\mathrm{K}}$, $k^{\mathrm{B}}$, $k^{\mathrm{A}}$). Even though it is possible to use custom quantization levels for the activations, its implementation needs special hardware in the datapath. Thus, the $k^{\mathrm{A}}$ is discarded, which better matches real-life platforms. The optimization problem we address in this work becomes

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{Q}}^{★}=& \underset{\mathcal{Q}}{argmax}\mathrm{accuracy}\left(\mathcal{W},\mathcal{Q};\mathcal{I}\right)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.c^{\mathrm{T}}<c^{\mathrm{T}-\max },c^{\mathrm{W}}<c^{\mathrm{W}-\max },c^{\mathrm{A}}<c^{\mathrm{A}-\max }\hfill \end{array}\end{array}$$

(4)

with

$$\mathcal{Q}=\left\{{\alpha }_i^{\{\mathrm{K},\mathrm{B},\mathrm{A}\}},b_i^{\{\mathrm{K},\mathrm{B},\mathrm{A}\}},k_i^{\{\mathrm{K},\mathrm{B}\}}\right\}$$

(5)

where $\mathcal{W}$ and $\mathcal{Q}$ are the model parameters and the quantization parameters of the CNN, respectively, and $\mathcal{I}$ is the validation data.

In this work, we want the most even results regardless of the quantization approach used; thus, a random search algorithm is used, even if the search is very slow. In [59], the search is described in detail.

4. Results

In this section, we will present the results obtained in two subsections: In Section 4.1, the exploration conducted across various cases is depicted, while Section 4.2 shows the outcomes of applying NUUQ to two typical use cases. In both scenarios, a downsized version of the VGG (See [59] for more detail) and the CIFAR10 dataset were employed.

For all cases, before optimizing quantization parameters ($\mathcal{Q}$), the constraints for the problem in (4) must be set. These maximum values for complexities are derived from a uniformly quantized model, enabling the standardization of these values. In other words, the numbers of bits and clusters for weights and activations (as applicable) are chosen to quantize a model uniformly and the resulting complexities of this model are used as the constraints in (10). This approach allows us to link a single tuple with number of bits and clusters, $(b^{\mathrm{K}},c^{\mathrm{K}},b^{\mathrm{B}},c^{\mathrm{B}},b^{\mathrm{A}})$ to complexity values, providing a more intuitive understanding for the reader. For instance, a uniformly quantized model with 4 bits and 4 clusters, $(b^{\mathrm{K}},c^{\mathrm{K}},b^{\mathrm{B}},c^{\mathrm{B}},b^{\mathrm{A}})=(4,4,4,4,4)$, has a given $(c^{\mathrm{T}},c^{\mathrm{W}},c^{\mathrm{A}})$. This allows us to claim that a model has been heterogeneously quantized with 4 bits and 4 clusters, which might have no meaning as the number of bits and clusters varies for each layer. However, it actually refers to the complexities of this model being less than or equal to those of the uniformly quantized model with these bit and cluster values. This provides a more intuitive way of expressing the complexities.

For the remaining figures in this section, when referring to the number of bits and clusters, it should be understood that there is a direct relationship with complexities, as explained in the previous paragraph.

4.1. Exploration

In this section exploration results are analysed. Observing the heatmaps in Figure 4, it is evident that accuracy is highly sensitive to both the number of activation bits and the average number of clusters. However, concerning the latter, it is also noticeable that once a certain threshold for accuracy is reached, the sensitivity diminishes. As for the number of bits, while there is a modest improvement as they increase, it is also noteworthy that accuracy shows little sensitivity to this value, at least within the range of 3 to 8 bits. On the other hand, the significance of applying FT is clear, as this process significantly enhances accuracy. This outcome is well-known, and therefore, not surprising. The application of FT typically depends solely on data availability.

For a deeper analysis, we consider the conditions under which 100% accuracy can be recovered in the context of NUUQ. These findings are segmented based on activation bits, the number of clusters, and the number of bits for weights, providing a comprehensive understanding of their respective impacts. The possibility of regaining full accuracy is significantly influenced by the value of $b^A$. Our analysis indicates a critical range between 4 and 6 bits for $b^A$, within which the feasibility of achieving 100% accuracy is determined, regardless of the application of FT. This value should be higher for the case without FT than in those with FT, but unfortunately it is not possible to define this value without further simulations. In the case of the number to recover full accuracy, the required value varies in the presence of FT and for different values of $b^A$.

Table 2. Approximate ranges for the number of clusters ($k^K$) required to achieve 100% accuracy in CNN quantization, categorized by activation bits ($b^A$) and the application of fine-tuning (FT).

Average Activation Bits (${\mathit{b}}^{\mathit{A}}$)	Number of Clusters
	Without FT	With FT
8	-	-
4	6.5 to 7.0	3.8 to 4.1
6	4.7 to 5.4	3.4 to 3.7

showcases this the ranges for $k^K$:

Our findings suggest that the specific value of $b^K$ is less critical, provided it exceeds 3 (lower values were not analysed). However, in scenarios with $b^A=4$ (both with and without FT) and $b^A=6$, an increase in $b^K$ appears to compensate for the limitations associated with lower $b^A$ or the absence of FT.

Understanding these critical values is not just academic; it is crucial for optimising CNN quantization. By identifying specific ranges and thresholds affecting accuracy recovery, we can define the parameter exploration space more effectively. This strategic approach tailors quantization parameters to meet accuracy requirements while optimising hardware resources. By avoiding values that lead to under-quantization, which wastes hardware resources, and over-quantization, which may lead to inadequate performance, the design space is smaller, enabling more efficient parameter exploration. These insights guide the quantization process, ensuring computational resources are allocated judiciously for optimal balance between accuracy and efficiency.

In

Table 3. Accuracies and complexities for noteworthy configurations and cases.

Case			Accuracy (SD)		Complexity
			Without FT	With FT	BOC	BWC	BAC
OM (FlP-32)			$0.852$ ($0.013$)	– (–)	NA	1.0	1.0
OM (FlP-16)			$0.852$ ($0.013$)	– (–)	NA	2.0	2.0
UQ-MP (FxP)	$b^{\mathrm{A}}$	$b^{\mathrm{K}}$	Without FT	With FT	BOC	BWC	BAC
	4	2	$0.422$  ($0.131$) $\downarrow \downarrow$	$0.644$  ($0.088$)  $\downarrow \downarrow$	2.4	16.0	29.3
		3	$0.791$  ($0.026$)  $\downarrow \downarrow$	$0.821$ ($0.016$) $\downarrow$	2.2	10.7	29.3
	6	2	$0.528$  ($0.078$)  $\downarrow \downarrow$	$0.751$  ($0.047$)  $\downarrow \downarrow$	2.2	16.0	19.5
		3	$0.815$ ($0.023$) $\downarrow$	$0.844$ ($0.010$) $\uparrow$	1.9	10.7	19.5
	8	2	$0.567$  ($0.085$)  $\downarrow \downarrow$	$0.765$  ($0.055$)  $\downarrow \downarrow$	2.0	16.0	14.7
		3	$0.825$ ($0.018$) $\downarrow$	$0.851$ ($0.009$) $\uparrow \uparrow$	1.7	10.7	14.7
NUUQ-MP (FxP)	$b^{\mathrm{A}}$	$b^{\mathrm{K}}$($k^{\mathrm{K}}$)	Without FT	With FT	BOC	BWC	BAC
	4	3 (3)	$0.713$  ($0.065$)  $\downarrow \downarrow$	$0.769$  ($0.059$)  $\downarrow \downarrow$	2.2	20.2	29.3
		8 (3)	$0.761$  ($0.028$)  $\downarrow \downarrow$	$0.807$ ($0.022$) $\downarrow$	1.5	20.2	29.3
		3 (4)	$0.807$ ($0.010$) $\downarrow \downarrow$	$0.829$ ($0.005$) $\downarrow$	2.2	16.0	29.3
		8 (4)	$0.812$ ($0.008$) $\downarrow$	$0.834$ ($0.004$) $=$	1.5	16.0	29.3
		4 (8)	$0.830$ ($0.003$) $\downarrow$	$0.845$ ($0.005$) $\uparrow$	2.0	10.7	29.3
		8 (8)	$0.831$ ($0.003$) $\downarrow$	$0.844$ ($0.005$) $\uparrow$	1.5	10.7	29.3
	6	3 (3)	$0.801$ ($0.018$) $\downarrow \downarrow$	$0.834$ ($0.019$) $=$	1.9	20.2	19.5
		8 (3)	$0.776$  ($0.042$)  $\downarrow \downarrow$	$0.828$ ($0.017$) $\downarrow$	1.2	20.2	19.5
		3 (4)	$0.835$ ($0.009$) $=$	$0.853$ ($0.008$) $\uparrow \uparrow$	1.9	16.0	19.5
		8 (4)	$0.837$ ($0.008$) $=$	$0.851$ ($0.007$) $\uparrow \uparrow$	1.2	16.0	19.5
		4 (8)	$0.852$ ($0.006$) $\uparrow \uparrow$	$0.861$  ($0.006$)  $\uparrow \uparrow$	1.7	10.7	19.5
		8 (8)	$0.853$ ($0.005$) $\uparrow \uparrow$	$0.864$  ($0.003$)  $\uparrow \uparrow$	1.2	10.7	19.5
	8	3 (3)	$0.797$   ($0.021$) $\downarrow \downarrow$	$0.846$ ($0.009$) $\uparrow$	1.7	20.2	14.7
		8 (3)	$0.798$  ($0.034$)  $\downarrow \downarrow$	$0.846$ ($0.008$) $\uparrow$	1.0	20.2	14.7
		3 (4)	$0.842$ ($0.005$) $\uparrow$	$0.858$ ($0.004$) $\uparrow \uparrow$	1.7	16.0	14.7
		8 (4)	$0.839$ ($0.005$) $=$	$0.856$ ($0.006$) $\uparrow \uparrow$	1.0	16.0	14.7
		4 (8)	$0.857$ ($0.003$) $\uparrow \uparrow$	$0.867$  ($0.003$)  $\uparrow \uparrow$	1.5	10.7	14.7
		8 (8)	$0.859$ ($0.003$) $\uparrow \uparrow$	$0.869$  ($0.004$)  $\uparrow \uparrow$	1.0	10.7	14.7

, some heatmap points are presented in numerical form. Based on these, some noteworthy results include the following:

• With NUUQ, it is possible to achieve accuracies above 99% (marked as ↑) with reductions of 10.7 times ($bA=4$), 16 times ($bA=6$), and even up to 20.2 times ($bA=8$).
• In some cases using NUUQ surpass the original accuracy (marked as $\uparrow \uparrow$) with reductions of 1.7 times ($bA=6$) and up to 16 times ($bA=8$).
• In all cases, comparing UQ-MP against NUUQ-MP shows that it is feasible to sacrifice some BOC for a significant improvement in BWC. For instance:

  −

  Compressed by 10.7 times, NUUQ-MP achieves improvements of up to 1.6% compared to UQ-MP.

  −

  Compressed by 16.0 times, NUUQ-MP achieves improvements of up to 9.3% compared to UQ-MP.

• NUUQ achieves compressions of around 20.2 times, losing only 0.6% accuracy for the bA=8 case. This compression level is equivalent to using approximately 1.58 bits for the quantization in terms of the BWC. These compression levels are prohibitive for UQ-MP.
• Regarding the BAC, there are no significant differences observed between UQ-MP and NUUQ-MP.

4.2. Use Cases

Let us explore two typical scenarios where using NUUQ quantization presents advantages. The first use case is common in a configurable iterative architecture where arithmetic is fixed, such as at 4, 8, or 16 bits. This situation is typically encountered in CNN implementations on FPGA, GPU, or CPU. While ASIC implementations are possible, they are less common since they would not fully leverage their advantages. In the second case, we consider unrolled architectures, typically found in both ASIC and FPGA. In such architectures, it is common to encounter different arithmetic applied in each layer.

Figure 5 and Figure 6 showcase the results for the first and second cases, respectively. In both scenarios, applying NUUQ demonstrates the potential for a reduction in BWC without sacrificing accuracy, although achieving this requires a trade-off in BOC. This is evident in Figure 5, where a NUUQ case without sacrificing BOC (blue curve) does not yield superior accuracy, as applying NUUQ without sacrificing BOC is equivalent to UQ.

In closing this exploration of use cases, we want to highlight how they apply to real situations. The examples we have talked about are based on real challenges that people face when using convolutional neural networks in edge devices [30,33,36,53]. We have shown how our NUUQ method can be adjusted and used effectively in different situations, which helps close the gap between research and actual use. In this way, our work is also useful for making AI a reality.

4.3. Comparison to Other Works

In Figure 7, we present a comparative analysis of the NUUQ strategy alongside methodologies from other studies. We compare the accuracy loss rate against the weight compression rate for a comprehensive evaluation across these works The metrics are defined as follows:

$$\mathrm{ACC}-\mathrm{rate}=\frac{\mathrm{ACC}}{\mathrm{ACC}-\mathrm{FlP}}\mathrm{BWC}-\mathrm{rate}=\frac{\mathrm{BWC}}{\mathrm{BWC}-\mathrm{FlP}}$$

(6)

here, ACC-FlP represents the accuracy of the original model using FlP arithmetic with 32 bits, while BWC-FlP denotes the BWC under the same conditions. Ideally, we aim for a high ACC-rate together with a low BWC-rate. Figure 7 shows these metrics for different number of activation bits, 4, 6 and 8, in Figure 7a, Figure 7b and Figure 7c respectively. Figure 7 aggregates all the previous cases alongside new values for the average activation bits. The effect of changing the average number of activation bits and the average number of bits used for weights (only for this work) can be seen as vertically aligned points, this is, points with the same BWC-rate but different ACC-rate. From this figure, it can be seen that our NUUQ approach consistently surpasses other methodologies in a broad range of scenarios with different average numbers of bits for activations and weights. However, it is important to acknowledge that comparisons across different studies are inherently complex and may not always be directly comparable due to various underlying factors:

• Unlike the method described in [18], our approach employs standard quantization for activations. This decision helps us avoid the need for specialized hardware in the datapath, ensuring better alignment with fundamental design principles. Nevertheless, exploring NUUQ for activations will be performed in future research to enhance design flexibility and enable comparative analysis.
• Our methodology inherently allows for more bits allocated to weights for a given BWC due to its unique ability to decouple the number of quantization levels from the bit count. This flexibility also means that for a fixed number of bits, our approach results in fewer quantization levels, making direct comparisons challenging.
• The CNN architectures employed vary across studies. Our analysis exclusively utilizes the VGG16 structure, while other research includes models like VGG, ResNet, and MobileNet, adding another layer of complexity to the comparison.
• Different datasets are used across different works. Our study utilizes CIFAR-10, whereas many others employ ImageNet. The choice of CIFAR-10 was made by the limitations of our search algorithm’s efficiency with larger datasets.
• Contrary to most studies that focus on search optimization, our work employs a random search strategy to ensure fairness and comprehensiveness in parameter exploration.
• Our FT methodology is applied globally across the entire network, diverging from other studies that FT sequentially layer by layer. While the latter may yield more refined FT results, our goal is to maintain fairness in parameter search.

These points reveal that our comparison is both detailed and complex, with some factors benefiting our approach and others presenting disadvantages.

4.4. Limitations

While the proposed NUUQ method demonstrates significant advancements in the efficient quantization of convolutional neural networks, particularly in terms of achieving high compression levels without substantial accuracy loss, it is important to recognize that are some limitations:

• Random search efficiency: The utilization of a random search algorithm for the optimization of quantization parameters, while effective in exploring a fair solution space, presents scalability challenges. This method’s computational intensity limits its applicability to larger network architectures or more extensive datasets.
• Network architecture scope: During our parameter exploration, we specifically focused on sequential CNN models because of their straightforward structure, which is conducive to clear and effective result demonstration. This choice enabled us to transparently showcase the advantages of our methodology. Nonetheless, it is important to note that concentrating on these simpler architectures may limit how our findings apply to more intricate or non-sequential network designs, such as those incorporating skip connections or parallel processing paths.
• Layer diversity: the current implementation of NUUQ has been applied to a select subset of neural network layers, primarily those commonly found in conventional CNNs used for classification tasks. The quantization techniques presented herein should be extended to other layer types.

By addressing these limitations in future iterations of our research, we aim to enhance the performance, applicability, and scalability of the NUUQ method.

5. Conclusions and Future Work

Firstly, the metrics proposed in this study provide more insightful information than classical metrics used in high-level models, making them more suitable for characterizing quantized models operating in FxP arithmetic.

Secondly, the key conclusion of our work is the notable advantages of applying NUUQ. The proposed technique introduces the number of clusters as a quantization parameter, providing an extra degree of freedom when applying mixed-precision quantization. Consequently, a trade-off relationship between BOC and BWC metrics is established, allowing for the possibility to sacrifice BOC for a reduction in BWC without facing a loss in accuracy. As a result, the number of potential quantization configurations increases, allowing for more efficient configurations.

Regarding future direction, we want to highlight a few aspects. Our immediate focus will be on refining the search methodology used for optimizing quantization parameters. The current random search approach, while comprehensive, limits the scalability of NUUQ to larger CNN architectures and more extensive datasets.

Another intriguing direction involves the quantization of activations in addition to weights and biases. Although this extension may pose challenges for hardware implementation due to the dynamic nature of activations, it holds potential for comprehensive comparisons and deeper insights into quantization impacts across different network components.

It is also important to address non-sequential networks, which will allow us to address a broader spectrum of CNN architectures, including those with complex topologies such as skip connections and parallel branches.

Incorporating depth-wise convolutions, known for their efficiency in CNN architectures, will be a significant step forward. Adapting NUUQ to support this layer type could enhance its utility, especially in mobile and embedded applications where computational resources are limited.

Finally, a vital aspect of our future work involves translating the theoretical and simulation-based outcomes of this research into real-world applications. Implementing NUUQ-optimized CNNs in hardware platforms such as ASICs, FPGAs, or microprocessors will validate its practical viability. The use cases presented in this study serve as a foundation for this transition, with the ultimate goal of deploying efficient, quantized CNNs in actual embedded systems.