A Method for In-Loop Video Coding Restoration

Abstract

In-loop restoration is a post-processing task aiming to reduce losses caused by the quantization and the inverse quantization phases in a video coding process. Emerging in-loop restoration methods, most of them based on deep learning, have reported higher quality gains than classical filters. However, the complexity at the decoder side remains a challenge. The Sparse Restoration Method (SRM) is presented as a low-complexity method that utilizes sparse representation and Natural Scene Statistic metrics to enhance visual quality at the block level. Our method shows potential restoration benefits when applied to synthetic video sequences.

1. Introduction

Video applications such as Ultra-High Definition (UHD), Virtual Reality (VR), online gaming, and social networks are increasingly growing. They were 65% [1] of the total Internet traffic in 2023. This trend leads current and emerging video codec standards such as High Efficiency Video Coding (HEVC) [2], AOMedia Video 1 (AV1) [3], Versatile Video Coding (VVC) [4], and AOMedia Video 2 (AV2) (evolution of existing AV1 and currently under standardization (https://gitlab.com/AOMediaCodec/avm (accessed on 1 December 2023)) to continue pushing boundaries of compression theory. All those codecs rely on the well-known hybrid video coding scheme [5], which utilizes block-based prediction, transform, and quantization coding. In addition to traditional video codecs, HEVC, AV1, VVC, and AV2, integrate an in-loop restoration block that aims to mitigate visual artifacts –such as blocking, ringing, and blurring– inevitably introduced during a compression process that causes degradation in both video coding efficiency and visual quality. In the aforementioned video codecs, the in-loop restoration is implemented through different approaches. In the case of AV1, the in-loop restoration relies on evaluating three options: Wiener filter, self-guided, and passthrough. Conversely, HEVC implements a Sample Adaptive Offset filter (SAO) and a deblocking filter (DBF). On the other hand, VVC and AV2, as future video codecs, are still evaluating new algorithms that increase coding efficiency and visual quality, for instance, the Convolutional Neural Network (CNN) methods introduced by Ding et al. [6] and Kong et al. [7]. Both approaches leverage the learning of distorted and clean frames, assisted by quantization compression level, to obtain superior restoration performance in terms of Bjontegaard Delta-Rate (BD-Rate) [8] against classic algorithms. However, those methods add a complexity factor of 30, at a decoder side, which is still out of the scope of future standards—a factor of less than or equal to 10.

In this context, we propose the Sparse Restoration Method (SRM) as a low-complexity in-loop restoration approach for AV2 aiming to increase video coding efficiency and visual quality while maintaining a decoder complexity in a factor of less that or equal to 10. SRM relies on the sparsity property of residuals to predict a sparse vector in a decoder. This prediction is used to compensate for each decoded (distorted) block. SRM models the decoding residuals (DR), calculated as the difference between a reference and a decoded frames, and utilizes the Discrete Cosine Transform (DCT) as a dictionary to exploit the statistical correlation of sparse vector nonzero coefficient magnitudes across different quantization levels. Additionally, a variable of the Natural Scene Statistic (NSS) model [9] is used as guiding information for blind in-loop restoration tasks at the block level. SRM is a threefold method: (1) sparse decoding residuals, (2) sparse coefficient magnitude estimator, and (3) sparse position estimator. We leveraged the official and open-access AOM raw video dataset to compare the performance of SRM against that of the anchor AV2 codec using three objective visual quality metrics: the Peak-Signal-to-Noise Ratio (PSNR), the Structured Similarity Indexing Method (SSIM) [10], and the Video Multi-Method Assessment Fusion (VMAF) (https://github.com/Netflix/vmaf (accessed on 1 June 2024)). Experimental evaluation shows that SRM performs in-loop restoration with a 1–2% BD-Rate gain under synthetic video sequences, indicating a potential use with cartoons, computer animated movies, and video games.

2. Related Works and Theoretical Background

2.1. In-Loop Restoration in Existing AV1 Video Codec

The reference video codec AV1 supports three post-processing filters that are enabled independently. The post-processing stage starts with a block resulting from the addition of prediction ($X_{pre}$) and residual ($\widehat{r}$) blocks. Then, the block is ready to fill in an inter-prediction buffer and pass to the display picture module. Before that, a post-processing filter may be applied with two main objectives: (1) improve inter-prediction for a more accurate reconstruction frame in the buffer and reduce overall bitrate, and (2) increase visual output quality. That is a typical case for a super-resolution, where before post-processing filters, there is an up-scaling operation that naturally truncates image details. Figure 1 presents the high-level workflow of a post-processing stage. Each filter is also further detailed.

2.1.1. Deblocking Filter

During the encoding process, a residual from reference (X) and predicted ($X_{pre}$) frames is split into transform blocks (TR). Each TR is computed using the discrete cosine transform (DCT), or the asymmetric discrete sine transform (ADST), to eliminate spatial correlation. Then, the resulting coefficients are quantized into N different levels determined by a quantization parameter (QP). The decoder performs inverse processes—transform and quantization—to recovers residuals, which are finally added to a predicted block ($X_{pre}$) to build a decoded block (Y). In most of the cases ($QP\ne 0$), inverse quantization and transformation introduce losses and boundary artifacts between adjacent transformation blocks (Figure 2). This effect occurs because the encoder processes each block independently (threads) and does not maintain smooth transitions at the boundaries. Therefore, AV1 implements a deblocking filter tool aiming to mitigate visual effects.

The deblocking algorithm uses vertical and horizontal lowpass FIR filters with 4, 8, or 14 taps for a luma component and 4 or 6 taps for a chroma component. The size of the lowpass FIR filters relies on the minimum transform size between blocks sharing a boundary. For instance, Figure 3 shows a case where the dimension of B-block determines the size of the filter. To avoid wrongly blurring natural edges, a deblocking tool implements a sequence of thresholds to determine if a filter is applied or not. The conditions are (1) $\mid p_1-p_0\mid >T_0$, (2) $\mid q_1-q_0\mid >T_0$, (3) $2*\mid p_0-q_0\mid +\frac{\mid p_1-q_1\mid }{2}>T_1$, (4) $\mid p_3-p_2\mid >T_0$, and (5) $\mid q_3-q_2\mid >T_0$, where $T_0,T_1>0$ and control the sensitivity of the algorithm. Conditions 4 and 5 are used only for filter taps 8 and 14.

Figure 4 illustrates the position of the pixels $q_x$ and $p_x$.

2.1.2. Constrained Directional Enhancement Filter (CDEF)

CDEF [11] is designed to remove ringing artifacts around hard edges—depicted in Figure 5. It is achieved by applying two filters (45° off) to each pixel. The selection of a proper filter is performed based on the minimization of Equation (1), as follows:

$$E_d^2=\sum _k\sum _{p\in P_{d,k}}{(x_p-{\mu }_{d,k})}^2,$$

(1)

where P is a group of pixels in a selected direction, as shown in Figure 6, and $\mu$ is the mean of the group P.

In-loop restoration filters are applied into loop restoration units (LRU) that can be $64\times 64$, $128\times 128$, or $256\times 256$ pixel blocks. Each LRU can independently select one of three possible restoration options.

2.1.3. Self-Guided

It relies on calculating two possible restored versions ($Y_1$ and $Y_2$) of a decoded block Y. Then, it projects a mismatch between each version and a reference to obtain a unified restored patch ($Y_r$), described in Equation (2).

$$Y_r=Y+\alpha (Y_1-Y)+\beta (Y_2-Y)$$

(2)

Before that, Y is denoised using Equation (3) on a pixel basis with two pairs of parameters sent from the encoder $(r_1,e_1),(r_2,e_2)$. Finally, $Y_1$ and $Y_2$ are computed using $\alpha$ and $\beta$. The self-guided filter requires four parameters to be exchanged from an encoder, as follows:

$$\widehat{y}=\frac{{\sigma }^2}{{\sigma }^2+e}y+\frac{e}{{\sigma }^2+e}\mu ,$$

(3)

where $\widehat{y}$ is a denoised pixel, $\mu$ is the mean of a block determined by r, and e is the standard deviation.

2.1.4. Wiener Filter

Wiener theory [13] is a well-known restoration technique widely applied to 1-D time series systems since 1949. It uses minimum mean square error (MMSE) to predict a signal $s\left(t\right)$, after being corrupted by noise $w\left(t\right)$. Both signals are considered wide-sense stationary processes. Wiener initially presented two versions: causal and non-causal filters. The last case was not physically realizable for time series because it considers past, present, and future samples.

Wiener’s theory became relevant but had not been applied into 2-D scenarios—image filtering, prediction, and smoothing. It was only in 1982 that Ekstrom [14] presented a physically realizable 2-D version of the original Wiener filter and demonstrated that it may also be extended to multi-dimensions. Ekstrom formulated an optimal error calculation to find the best parameters that define a filter configuration as described in the following Equation (4):

$$H\left(z\right)=\frac{S_{xy}\left(z\right)}{S_{xx}\left(z\right)},$$

(4)

where $S_{xy}$ is the spectral energy of the correlation between X (reference) and Y (distorted) images. Similarly, $S_{xx}$ is the spectral energy of the autocorrelation of X.

The feasibility of applying Wiener filter on images gained relevance in video restoration problems, since it aims to find a kernel that linearly relates an original image and its distorted version. In other words, it allows an encoder to determine kernel coefficients that a decoder can use to restore each block. This leads to the implementation of Wiener in both AV1 and HEVC, and it is still part of AV2. The first version implemented in AV1 presented outstanding compression efficiency, but complexity and signal information were desirable for improvement. Therefore, under the assumption of symmetry, Siekmann et al. [15] proposed a separable filter that achieves reductions of 33.33% and 50% in sum and multiplication operations for a $3\times 3$ block. In addition, for a $9\times 9$ block size, efficiencies of 77.77% and 80% were observed for sum and multiplications, respectively.

Besides operation performance, signaling information is also drastically reduced. For example, a $9\times 9$ block using a non-separable filter requires 81 coefficients. For a non-separable symmetric filter, it requires 41 coefficients; and in the case of a separable symmetric filter, it requires 18 coefficients. In other words, it represents 78% less signaling information to send to an encoder. It was also noted that non-separable and separable filters maintain a similar performance in terms of bit saving. The separable version is, in fact, the official implementation of Wiener filter in an AV1 reference code. However, processing time and complexity are still factors to improve in future approaches.

Table 1. Performance results of Wiener filter (on) vs. passthrough (off) mode in AV1 [3].

	On	Off	Perf. On vs. Off
Average speed (fps)	15	22	$-32$%
Total encoding time (ms)	40,856	27,951	$+46$%

presents the performance results of Wiener filter vs. passthrough mode using AV1 over 10 frames ($1920\times 1080$).

The post-processing stage starts with a block (Y) resulting from the addition of prediction ($X_{pre}$) and residual ($\widehat{r}$) blocks. Then, Y is ready to fill in an inter-prediction buffer and pass to the display picture module. Before that, a post-processing filter may be applied with two main objectives: (1) improve inter-prediction for a more accurate reconstruction frame in the buffer and reduce the overall bitrate and (2) increase the visual output quality. Figure 1 presents the high-level workflow of the post-processing stage. In-loop restoration is part of the post-processing and aims to find the coefficients of a filter, either self-guided or Wiener, which increases the visual quality when applied to the distorted block—received by the decoder. An in-loop filter is also designed to achieve the minimum signaling bits required to transmit the coefficients from an encoder to a decoder. In the case of the Wiener filter, Siekmann et al. [15] proposed a separable version that requires 78% less signaling information bits compared with the original non-separable Wiener method. All post-processing modules, including in-loop filters, are included in the baseline version of the AV2 video codec that is still in the standardization process.

2.2. In-Loop Restoration Based on Deep Learning

In-loop restoration approaches for tackling video compression problems from the perspective of deep learning architectures become more limited and point to video coding applications in most cases. At first, Jia et al. [16] proposed a content-aware CNN-based (CAC) method based on a block-based model selection and restoration modules incorporated into the HEVC reference code. The first component implements a discriminative network to select the most appropriate CNN [17] per Coding Tree Unit (CTU). The selection, of which CNN is used, relies on the previous labeling of a training content, where several categories are considered. Therefore, the discriminative network chooses a CNN, which minimizes a loss function described in Equation (6), considering a specific content, as follows:

$$J_{CTU}=\Delta D_{CTU}+\lambda R,$$

(5)

$$\Delta D_{CTU}=D_{CTU}^{\prime }-D_{CTU},$$

(6)

where $J_{CTU}$ refers to the performance of the n-sima CNN, ${\Delta }_{CTU}$ is the variation in terms of BD-rate between a CTU before and after the restoring process, $\lambda$ is the Lagrange multiplier that controls the trad-off between rate and distortion. Finally, R represents the required amount of bits for signaling. The CAC method has been evaluated against anchor HEVC, obtaining performance improvements in BD-rate between 2% and 4%. Complexity is also considered against Very Deep Super Resolution (VDSR) [18], Variable-Filter-Size Residue-Learning CNN (VRCNN) [19], and Adaptive Loop Filtering (ALF) [20]. Reported results indicated that the ALF—a current non-DL-based algorithm embedded into HEVC—is still more efficient achieving 123% vs. 11656% of CAC in terms of decoder complexity.

Considering previous results, it is straightforward that quality may be enhanced. However, complexity is still far from reaching the expected level of future video codec standards (<10×). Inspired by these facts, Ding et al. [6] proposed SimNet, a CNN-based method that reduces complexity by leveraging a skipping strategy where a simple CNN network restores specific frames and the remaining frames continue using traditional restoration non-DL-based methods. SimNet has been formally evaluated to be a part of AV2. It applies restoration on intra and Inter frames. The Inter frames rely on a propagation impact over the group of pictures (GOP). SimNet contains N cascading convolutional layers and a ReLu at last (Figure 7). The depth of the networks depends on a QP (Quantization Parameter) at each block.

SimNet reported overall performance improvements in BD-rate of 7.27% and 5.47% for Intra and Inter frames, respectively, against the reference codec AV1. It also showed a processing time reduction of 12.65% compared with AV1. However, the original paper does not provide a report of the decoder complexity.

The original publication of SimNet highlights an over-filtering problem closely related to an effect caused by Inter frames that are filtered and then used as a reference for next frames. The effect is a propagated noise that reduces the performance of CNN-based restoration methods in comparison with AV1. SimNet tackles this issue by skipping frames that potentially introduce errors. Considering this problem, Ding et al. [21] propose a transfer-learning method for only-Inter frames that incorporates back reconstructed frames into a training set of the CNN model. The algorithm achieves higher visual quality against the reference code HEVC but increases the processing time by around 4%. The encoder complexity is not reported, but we infer that SimNet may consume around 5x due to progressive training.

Kong et al. [7] present a Guided CNN architecture with similar outcomes, BD-rate: 1.84–3.06% and additional processing time, 23.79% (Figure 8). The restoration lies on a linear combination of N-CNN’s weighted outputs. Each CNN aims to capture distinct features of an image. However, the paper does not specify which characteristics capture from an image. The idea behind this is to have a lightweight array of N-CNN and obtain an optimal vector of weights that minimize the loss function described in Equation (7).

$$r_{corr}=a_0{r}_0+a_1{r}_1+...+a_{M-1}{r}_{M-1}.$$

(7)

3. Sparse Restoration Method (SRM)

Considering opportunities in the field of in-loop restoration, we introduce SRM that is threefold: (1) sparse decoding residual, (2) sparse coefficient magnitude estimator, and (3) sparse position estimator.

3.1. Sparse Decoding Residuals

During the video compression process, spatial residuals—the difference between a reference and an Inter/Intra predicted frames—are transformed into the frequency domain. The resulting coefficients, such as DCT, are mapped into predefined quantized factors, which may vary from 0 to 255, where 0 is lossless. After that, an entropy operation is applied to eliminate statistical redundancies. In the particular case of DCT, it means using more bits to represent DC and low-frequency coefficients. Finally, a sequence of bits is sent to the decoder. Equation (8) models frequency domain residuals after quantization Q [22] and transform T operations, as follows:

$$g=Q\left[T\right\{r\left\}\right].$$

(8)

In order to recover an original frame, the decoder uses the prediction frame and its corresponding residual. The latter is obtained after applying inverse transform and quantization operators to an entropy-decoded bitstream. However, quantization is a lossy task that depends on the number of levels—such as 85, 110, or 210. A trade-off between bitrate and quality is expected. Therefore, the result of an inverse transform is not equal to a residual r (Equation (9)), as follows:

$$\widehat{r}=T^{-1}\left[Q^{-1}\left\{g\right\}\right],$$

(9)

$$r\ne \widehat{r}.$$

Since quantization error ($e_q$) is linear, we can approximate r as follows:

$$r\approx T^{-1}[Q^{-1}\left\{g\right\}+e_q],$$

(10)

$$r\approx \widehat{r}+T^{-1}\left[e_q\right].$$

As illustrated in Figure 1, a reference frame X is theoretically expected to be equal to its prediction $X_{pre}$ plus a residual r. Integrating this into Equation (10) leads to the expression of a relation between a decoded frame Y and its reference X in the following Equation (11):

$$X\approx X_{pre}+r,$$

(11)

$$X\approx X_{pre}+\widehat{r}+{\mathbf{T}}^{-\mathbf{1}}\left[{\mathbf{e}}_{\mathbf{q}}\right],$$

$$X\approx Y+{\mathbf{T}}^{-\mathbf{1}}\left[{\mathbf{e}}_{\mathbf{q}}\right].$$

At the decoder, $X_{pre}$ and $\widehat{r}$ are always obtained through prediction and inverse quantization/transformation operations, respectively. Therefore, SRM models a differential cause by the inverse transform of a quantization error (${\mathbf{T}}^{-\mathbf{1}}\left[{\mathbf{e}}_{\mathbf{q}}\right]$). From now on, we call the differential decoding residual (DR). The reference AV1 video codec implements a series of tools to reduce the effect of quantization on block boundaries or recover lost information—the same as DR. Those processes require few bytes to improve the visual quality of a decoded frame, and the final bitrate is barely impacted. SRM takes into account the constraint of a low impact in the final bitrate by modeling DR using sparse theory to rely on a few nonzero coefficients expanded by a proper basis—or a dictionary.

In our proposal, we use $\gamma$ to represent DR, a sparse vector $\alpha$, a dictionary $\Phi$, and the $L_0$—norm to state the problem in Equation (12):

$${\left|\right|\alpha \left|\right|}_0\le zs.t.\gamma =\alpha \Phi .$$

(12)

The use of an $L_0$ norm and the Orthogonal Matching Pursuit [23] algorithm relies on our target, which is to minimize the total of nonzeros instead of an error. The solution of the optimization problem, in Equation (12), provides a sparse vector $\alpha$ that, together with a dictionary $\Phi$, can estimate a residual $\gamma$. The calculation of $\alpha$ is run entirely on the encoder. The resulting few nonzero coefficients and the corresponding positions in a vector are sent to the decoder as restoration signaling information. The residual $\gamma$ is split into $\sqrt{n}\times \sqrt{n}$ (i.e., $8\times 8$, $16\times 16$, or $32\times 32$ blocks), which are processed simultaneously to make the problem suitable in terms of computer memory usage. An operator $R_{i,j}$ is introduced to extract a $i,j$ patch of size $\sqrt{n}\times \sqrt{n}$. Equation (13) models $i,j$ sparse vector $\alpha$, as follows:

$$\left|\right|{\alpha }_{i,j}{\left|\right|}_0\le z,s.t.R_{i,j}\gamma ={\alpha }_{i,j}\Phi .$$

(13)

The patch-based approach requires blocks to be overlapped in order to avoid edge reconstruction artifacts. Mairal et al. [24] introduce a weighted average formulation that we adapt in the following Equation (14):

$$\widehat{\gamma }={\left(\sum _{i,j}{R}_{i,j}^T{R}_{i,j}\right)}^{-1}\left(\sum _{i,j}{R}_{i,j}^T\Phi {\alpha }_{i,j}\right).$$

(14)

3.2. Sparse Coefficient Magnitude Estimation

The statistical redundancy of the sparse coefficients is exploited using a DCT basis as a dictionary. Thus, a sparse vector is interpreted as a truncated version of $T\left[\gamma \right]$, where T is the DCT transform operator. Therefore, nonzero coefficients follow a predictable statistical behavior [25]. In fact, AV1/AV2 models the magnitude of DCT coefficients as the Laplace distribution [26], which permits the estimation of rate distortion per different transform configurations. Our method uses a similar assumption, but with the Gaussian distribution Equation (15), since it fits better the empirical distribution of the absolute magnitude of a DC nonzero coefficient (z) and the Gamma distribution Equation (16) for the rest of the coefficients. The only exception is $QP=85$, where the Laplace distribution Equation (17) better fits the empirical distribution of the absolute value of the AC coefficients.

$$f(\mid z\mid )=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{1}{2}{\left(\frac{\mid z\mid -\mu }{\sigma }\right)}^2},$$

(15)

$$f(z\mid a,b)=\frac{z^{a-1}{e}^{-bz}{b}^a}{\Gamma \left(a\right)},$$

(16)

$$f(z\mid \mu ,b)=\frac{1}{2b}{e}^{\left(-\frac{\mid z-\mu \mid }{b}\right)}.$$

(17)

The parameters of the distributions described in Equations (15)–(17) are correlated with the QP levels. Accordingly, we estimate those parameters per each $QP\in \{85,110,135,160,185,210\}$, as shown in the following

Table 2. Estimated parameters of the probability distribution functions by QP level.

QP	DC	AC
85	$--$	$Laplace(\mu =6.01,b=1.08)$
110	$\mathcal{N}(a=13.25,b=0.70)$	$\mathcal{N}(\mu =10.06,\sigma =2.77)$
135	$\Gamma (a=6.30,b=2.42)$	$\mathcal{N}(\mu =17.98,\sigma =6.94)$
160	$\Gamma (a=4.09,b=5.89)$	$\mathcal{N}(\mu =33.45,\sigma =14.68)$
185	$\Gamma (a=3.04,b=13.11)$	$\mathcal{N}(\mu =61.37,\sigma =28.53)$
210	$\Gamma (a=2.50,b=27.81)$	$\mathcal{N}(\mu =110.20,\sigma =54.29)$

.

SRM estimates nonzero coefficients following the probability distribution functions instead of sharing them with a decoder. The magnitude of a coefficient may vary but still adds gain to a distorted frame, since the method is concentrated in a residual, not in a frame, as illustrated in Figure 9. For instance, in $QP=135$, the cost of predicting coefficients is −0.16 dB in terms of PSNR. This means that the decoder compromises 0.16 dB in quality but, at the same time, achieves an approximately 25 K (2.78%) byte reduction per frame of dimensions $1280\times 720$ (This example use blocks of $16\times 16$ and ${\parallel \alpha \parallel }_0=2.$)

3.3. Sparse Coefficient Position Estimation

The positions of nonzero coefficients, in $\alpha$, are most relevant frequencies in a decoded block Y, which requires compensation, from a frequency domain perspective. On the decoder side, this information is unknown. However, assuming that quantization errors equally affect all AC components of Y, we expect that the most relevant frequencies in Y are also the ones requiring more restoration. Experimental validation—by executing restoration over patches across raw video sequences with different QP levels—shows that in 80% of the cases (across all QP levels), at least one of the three most relevant coefficients in Y corresponds to the most relevant frequencies in the residual decoding block. Thus, implementing a DCT transform of Y could bring this information to the decoder. However, the sign of DCT coefficients does not keep a relation with a potential sign of the nonzero coefficient on a specific position. Therefore, we introduced a novel algorithm that relies on image quality blind assessment in the frequency domain, introduced by Saad M. et al. [27], to evaluate the most efficient combination, in terms of the sign of two nonzero coefficients in the sparse vector whose positions are determined by the location of the most relevant components of a DCT transform of Y.

Figure 9. Visual and objective comparison of a frame restoration. At the top, sparse coefficients are sent to the decoder. At the bottom, sparse coefficients are predicted by the decoder. The distorted original image (not displayed here) reports a PSNR = 38.81 dB. This example uses $QP=135$.

Figure 9. Visual and objective comparison of a frame restoration. At the top, sparse coefficients are sent to the decoder. At the bottom, sparse coefficients are predicted by the decoder. The distorted original image (not displayed here) reports a PSNR = 38.81 dB. This example uses $QP=135$.

We follow a well-established definition of NSS where natural images are characterized by a Generalized Gaussian Distribution (GGD) and the effects of distortions, such as JPEG blocking and blur, affect the shape and histogram of an original image. However, we are not interested in evaluating image quality. Instead, we use features of GGD as criteria to determine a proper predicted decoding residual block. Specifically, we utilize the GGD constant parameter a described in Equation (18). This allows us to avoid signaling data between the encoder and the decoder to share coefficients information, besides a single bit per restoration block to guide a restoration process. The complete details of our prediction algorithm are given below in Algorithms 1 and 2.

$$F_X(x,\mu ,{\sigma }^2,\tau )=a{e}^{{(-b\mid x-\mu \mid )}^{\tau }}x\in \Re ,$$

(18)

$$b=\left(\frac{1}{\sigma }\sqrt{\frac{\Gamma (3/\varphi )}{\Gamma (1/\varphi )}}\right)a=\left(\frac{b\varphi }{2\Gamma (1/\varphi )}\right).$$

Algorithm 1 Sparse prediction algorithm at the encoder per block basis.

Input: Reference block: x, Decoded block: y, block size: ($\sqrt{n}\times \sqrt{n}$)

Task: Set encoder-flag

Apply DCT to x and y $x^f=T\left[x\right],y^f=T\left[y\right]$ Calculate the GGD feature a for x and y: $a_x=\left(\frac{b_x\varphi }{2\Gamma (1/\varphi )}\right)a_y=\left(\frac{b_y\varphi }{2\Gamma (1/\varphi )}\right)$ Set encoder-flag: $\frac{a_x}{a_y}>1\to$ encoder-flag=1, $\frac{a_x}{a_y}<1\to$ encoder-flag=0

Algorithm 2 Sparse prediction algorithm at the decoder.

Input: Decoded block: y, block size: ($\sqrt{n}\times \sqrt{n}$), $encoder\_flag$ $\in \{0,1\},QP$

Task: Predict the decoding residual block at the decoder ($\gamma$)

Apply DCT to the decoded block: $y^f=T\left[y\right]$ Identify the position in $y^f$ of the two coefficients with the largest absolute value: $A=sort(\mid y^f\mid )s.t.max\left(A\right)=A\left[0\right]\wedge A\left[0\right]\ge A\left[1\right],$ $p_0=k_{0}s.t.A\left[0\right]=\mid y_{k_0}^f\mid ,$ $p_1=k_{1}s.t.A\left[1\right]=\mid y_{k_1}^f\mid$ Predict the magnitude of the sparse nonzero coefficients: $\{c_0,c_1\}\sim Laplace(\mu ,b)\forall QP\in \left\{85\right\},$ $\{c_0,c_1\}\sim \mathcal{N}(\mu ,{\sigma }^2)\forall QP\in \{110,135,160,185,210\}$ Create four sparse vectors with the possible sign combinations: $\forall i\notin \{p_0,p_1\}{\nu }_0\left[i\right]=0,{\nu }_0\left[p_0\right]=+c_0,{\nu }_0\left[p_1\right]=+c_1,$ $\forall i\notin \{p_0,p_1\}{\nu }_1\left[i\right]=0,{\nu }_1\left[p_0\right]=-c_0,{\nu }_1\left[p_1\right]=+c_1,$ $\forall i\notin \{p_0,p_1\}{\nu }_2\left[i\right]=0,{\nu }_2\left[p_0\right]=+c_0,{\nu }_2\left[p_1\right]=-c_1,$ $\forall i\notin \{p_0,p_1\}{\nu }_3\left[i\right]=0,{\nu }_3\left[p_0\right]=-c_0,{\nu }_3\left[p_1\right]=+c_1$ Obtain four potential restored blocks in the DCT domain: $y_0^f=y^f+{\nu }_0,$ $y_1^f=y^f+{\nu }_1,$ $y_2^f=y^f+{\nu }_2,$ $y_3^f=y^f+{\nu }_3$ Calculate the GGD feature a for each potential restored block: $b_i=\left(\frac{1}{{\sigma }_i}\sqrt{\frac{\Gamma (3/\varphi )}{\Gamma (1/\varphi )}}\right),a_i=\left(\frac{b_i\varphi }{2\Gamma (1/\varphi )}\right),$ $A=\{a_0,a_1,a_2,a_3\}$ Select the restored block that obtains the max or min of the GDD feature a: encoder-flag=0 $\to \{a_i=\max \left(A\right)\},$ encoder-flag=1 $\to \{a_i=\min \left(A\right)\}$ $\gamma ={\nu }_i\Phi ,y_{resto}=y+\gamma$

4. Experiments Design

4.1. Dataset

For the experimentation, we follow the AOM Common Test Conditions v2.0 (https://aomedia.org/docs/CWG-B075o_AV2_CTC_v2.pdf (accessed on 1 December 2023)), which recommends a series of raw video sequences with diverse characteristics—including content type, bit depth, resolution, and color sub-sampling—intending to provide standard scenarios to test new algorithms against the reference codec. Thus, five test sequences were selected to assess the performance of SRM. The selection was made based on two criteria: (1) sequences with content proper for All Intra (AI) configuration, and (2) due to restrictions in cloud computing services, we choose sequences with resolutions < 1920 × 1080 and bit-depth = 8.

Table 3. Selected raw video test sequences.

Class	Sub-Class	Resolution	Total
Natural Videos (A)	A2	$1920\times 1080$	10
	A3	$1280\times 720$	6
	A4	$640\times 360$	6
	A5	$480\times 270$	3
Synthetic (B)	B1	$1920\times 1080$	7

presents general details of a selected sequences. In addition, the standardization group defines a set of QPs to evaluate each configuration. In our case, we follow All Intra(AI) recommendation: $QP=\{85,110,135,160,185,210\}$. Raw videos are publicly available and hosted at an open-source platform: Xiph (https://media.xiph.org/video/aomctc/test_set/ (accessed on 1 January 2024)). Figure 10 shows examples of frames belonging to sub-classes A2–A5 and B1.

4.2. Computing Details

Considering the total number of video sequences (32) and the number of QP levels (6), we conducted 192 executions using the anchor encoder configuration (AV2 + switchable filter) (https://gitlab.com/AOMediaCodec/avm (accessed on 2 February 2024)) and 192 executions using SRM. At each execution, we also performed a post-processing reference quality assessment using PSNR, SSIM, and VMAF. In all cases, compressed file and quality metrics are automatically uploaded to an AWS S3 bucket as a source to the overall performance evaluation based on the BD-Rate. Each of the 384 tests was performed using AWS Batch (https://aws.amazon.com/batch/ (accessed on 4 February 2024)), leveraging a docker container and automation, as described in the reference architecture for a super-resolution [28], as illustrated in Figure 11. In addition, we use EC2 instance type c5.2xlarge as a part of our AWS Batch job definition.

5. Results

Table 4. AV2 + SF (switchable filter) vs. AV2 + SRM (our method). BD-Rate (PSNR, SSIM, and VMAF) for Only-Intra mode is assessed on sequences A2–A5 and B1. AV2 + SRM surpasses the baseline method (AV2 + SF) in synthetic videos (B1).

		BD-Rate *
Sequence	Implementation	PSNR	SSIM	VMAF
A2	AV2 + SF	−1.816	−0.035	−1.994
	AV2 + SRM	0.487	−0.337	−2.206
A3	AV2 + SF	−1.813	−0.183	−2.310
	AV2 + SRM	0.794	0.763	−1.642
A4	AV2 + SF	−2.156	−1.890	−0.326
	AV2 + SRM	1.158	1.355	0.730
A5	AV2 + SF	−0.499	−0.627	−1.33
	AV2 + SRM	0.615	−0.174	−1.276
B1	AV2 + SF	−0.337	0.025	−1.603
	AV2 + SRM	1.047	−0.949	−2.585

* Negative is better.

presents a comparison of SRMs across video sequences A2–A5 and B1. We use the BD-Rate to evaluate the performance under different bitrate and quality conditions. Despite the prediction efficiency, with regard to the accuracy and reduction of signaling bits, the encoder has to inform the decoder whether a block is subjected to restoration or not. SRM is partially blind, which can restore block accuracy in most cases (up to 70%). In the remaining 30%, the encoder has to inform the decoder that restoration is not required. A guide bit (encoder-flag) is also added to patches that are subjected to restoration. Therefore, a frame from group A2 ($1920\times 1080$) with a block size of $32\times 32$ requires approximately 379 bytes for signaling, considering that 50% of blocks are restored. This number (379 bytes) is double that of the switchable filter.

Another exciting result is that SRM performs better on SSIM and VMAF metrics for B1 sequences, which, according to the MSU Graphics and Media Lab Video Group [29], are 90.57% and 93.86% (respectively) correlated to human subjective score, which surpasses the 87.43% reported for PSNR. It goes in the direction of NSS, which assesses the structure of images instead of measuring a distance between pixels. Additionally, SSIM and VMAF use the principle of structure of images. Figure 12 presents a subjective example of the performance of SRM restoration. In this case, the restored block starts to recover details compared with the decoded block. In a similar case, in Figure 13, we observed how the bush in the image recovers part of its original shape that was lost during the quantization and inverse quantization processes. Our method aims to compensate frequencies—in DCT-domain—that are more relevant for an image and provide more gain in terms of details. Therefore, we omit DC coefficients that are not considered by a GGD feature.

Regarding complexity, SRM shows approximately 110% and 115% processing time per frame of natural (A2–A5) and synthetic (B1) sequences compared with anchor AV1/AV2 + SF (

Table 5. Only-Intra computational complexity (seconds/frame).

Sequence	AV2 + SF *	AV2 + SRM *	$\mathsf{\Delta }$ Time
A2–A5	18.32	20.21	+110.31%
B1	14.84	15.61	+105.18%

* Running on AWS c5.2xlarge instance.

). Those results are aligned with the low-complexity implementation of sparse analysis against traditional deep learning methods detailed in Section 2.2.

6. Conclusions and Future Works

• We proposed SRM as a low-complexity method that shows the capability of being used with synthetic video sequences. A large market of digital games, child movies, and educational and training videos, among others, can benefit from the use of SRM.
• SRM was able to predict a proper DR (decoding residual) block using the GGD shape parameter (a) as a quality selector factor at a block level in the DCT space, which maximized the objective visual quality of a restored block. Moreover, SRM achieved a 2.5 BD-rate gain, in terms of VMAF, against existing switchable restoration filter—in AV1/AV2—over synthetic content (class B), while complexity was kept between 105% and 110%. SRM leverages a guiding restoration flag to determine which blocks require restoration and which blocks can be passthrough. A future improvement for SRM will be targeted to predict this flag at the decoder side, which will reflect a significant bitrate reduction (approx. 10–15%).
• Sparse representation is, without a doubt, an efficient method for image restoration tasks. Regarding the video coding in-loop restoration scenario, the critical challenge was eliminating required information to transfer between the encoder and decoder to represent the nonzero coefficients. However, moving the high-intensive task to the decoder is not an option, considering the real-time exigency during the decoding process. Therefore, we developed a hybrid approach where most of the required information was predicted in the decoder, and only a guiding bit encoder-flag was required. The reason for utilizing a guiding bit is the poor precision of predicting if a block, i.e., $32\times 32$, should be collapsed or expanded in terms of the GGD.
• Traditional full-reference quality metrics, such as PSNR, SSIM, and VMAF, are not completely consistent for assessing image/video restoration. These metrics rely on an existing reference image that is not always artifacts free—i.e., noise, blocking, and blurring. Therefore, future works should be addressed to define computational-efficient and real-time non-reference metrics at a frame and block levels, in order to provide human-correlated data to the decoder to perform restoration without requiring context information shared by the encoder. This mechanism will improve in-loop restoration efficiency in terms of the required amount of signaling bits.