Fast Flow Reconstruction via Robust Invertible n × n Convolution

Abstract

Flow-based generative models have recently become one of the most efficient approaches to model data generation. Indeed, they are constructed with a sequence of invertible and tractable transformations. Glow first introduced a simple type of generative flow using an invertible $1\times 1$ convolution. However, the $1\times 1$ convolution suffers from limited flexibility compared to the standard convolutions. In this paper, we propose a novel invertible $n\times n$ convolution approach that overcomes the limitations of the invertible $1\times 1$ convolution. In addition, our proposed network is not only tractable and invertible but also uses fewer parameters than standard convolutions. The experiments on CIFAR-10, ImageNet and Celeb-HQ datasets, have shown that our invertible $n\times n$ convolution helps to improve the performance of generative models significantly.

1. Introduction

Supervised deep learning models have recently achieved numerous breakthrough results in various applications, for example, Image Classification [1,2,3], Object Detection [4,5,6], Face Recognition [7,8,9,10,11,12,13,14], Image Segmentation [15,16] and Generative Model [17,18,19,20,21,22]. However, these methods usually require a huge number of annotated data, which is highly expensive. In order to tackle the requirement of large annotations, generative models have become a feasible solution. The main objective of generative models is to learn the hidden dependencies that exist in the realistic data so that they can extract meaningful features and variable interactions to synthesize new realistic samples without human supervision or labeling. Generative models can be used in numerous applications such as anomaly detection [23], image inpainting [24], data generation [20,25], super-resolution [26], face synthesis [22,27,28], and so forth. However, learning generative models is an extremely challenging process due to high-dimensional data.

There are two types of generative models extensively deployed in recent years, including likelihood-based methods [29,30,31,32] and Generative Adversarial Networks (GANs) [33]. Likelihood-based methods have three main categories: Autoregressive models [30], variational autoencoders (VAEs) [34], and flow-based models [29,31,32]. The flow-based generative model is constructed using a sequence of invertible and tractable transformations, the model explicitly learns the data distribution and therefore the loss function is simply a negative log-likelihood.

The flow-based model was first introduced in [31] and later extended in RealNVP [32]. These methods introduced an affine coupling layer that is invertible and tractable based on Jacobian determinant. As the design of the coupling layers, at each stage, only a subset of data is transformed while the rest is required to be fixed. Therefore, they may be limited in flexibility. To overcome this limitation, coupling layers are alternated with less complex transformations that manipulate on all dimensions of the data. In RealNVP [32], the authors use a fixed channel permutation using fixed checkerboard and channel-wise masks. Kingma et al. [29] simplifies the architecture by replacing the reverse permutation operation on the channel ordering with invertible $1\times 1$ convolutions.

However, the $1\times 1$ convolutions are not flexible enough in these scenarios. It is extremely hard to compute the inverse form of the standard $n\times n$ convolutions, and this step usually produces high computational costs. There are prior approaches that design the invertible $n\times n$ convolutions by using emerging convolution [35], periodic convolutions [35], autoregressive flow [36] or stochastic approximation [37,38,39]. In this paper, we propose an approach to generalize an invertible $1\times 1$ convolution to a more general form of $n\times n$ convolution. Firstly, we reformulate the standard convolution layer by shifting the inputs instead of the kernels. Then, we propose an invertible shift function that is a tractable form of Jacobian determinant. Through the experiments on CIFAR-10 [40], ImageNet [41] and Celeb-HQ [42] datasets, we prove that our proposals are significant and efficient for high-dimensional data. Figure 1 illustrates the advantages of our approach with high-resolution synthesized images.

Contributions: This work generalizes the invertible $1\times 1$ convolution to an invertible $n\times n$ convolution by reformulating the convolution layer using our proposed invertible shift function. Our contributions can be summarized as follows:

• Firstly, by analyzing the standard convolution layer, we reformulate its equation into a form such that, rather than shifting the kernels during the convolution process, shifting the input provides equivalent results.
• Secondly, we propose a novel invertible shift function that mathematically helps to reduce the computational cost of the standard convolution while keeping the range of the receptive fields. The determinant of the Jacobian matrix produced by this shift function can be computed efficiently.
• Thirdly, evaluations of several datasets on both objects and faces have shown the generalization of the proposed $n\times n$ convolution using our proposed novel invertible shift function.

2. Related Work

The generative models can be divided into two groups, that is, Generative Adversarial Networks and Flow-based Generative Models. In the first group, Generative Adversarial Networks [33] provide an appropriate solution to model the data generation. The discriminative model learns to distinguish the real data from the fake samples produced using a generative model. Two models are trained as they are playing a mini-max game. Meanwhile, in the second group, the Flow-based Generative Models [29,31,32] are constructed using a sequence of invertible and tractable transformations. Unlike GAN, the model explicitly learns the data distribution $p\left(\mathbf{x}\right)$ and therefore the loss function is efficiently employed with the log-likelihood.

In this section, we discuss several types of flow-based layers that are commonly used in flow-based generative models. An overview of several invertible functions is provided in the

Table 1. Comparative invertible functions in several generative normalizing flows.

Description	Function	Reverse Function	Log-Determinant
ActNorm [29]	$\mathbf{y}=\mathbf{x}\odot \gamma +\beta$	$\mathbf{x}=(\mathbf{y}-\beta )/\gamma$	$\sum log\left|\gamma \right|$
Affine Coupling [32]	$\mathbf{x}=[{\mathbf{x}}_a,{\mathbf{x}}_b]$	$\mathbf{y}=[{\mathbf{y}}_a,{\mathbf{y}}_b]$	$\sum log\left|s\right({\mathbf{x}}_b\left)\right|$
	${\mathbf{y}}_a={\mathbf{x}}_a\odot s\left({\mathbf{x}}_b\right)+t\left({\mathbf{x}}_b\right)$	${\mathbf{x}}_a=[{\mathbf{y}}_a-t\left({\mathbf{y}}_b\right)]/s\left({\mathbf{y}}_b\right)$
	$\mathbf{y}=\left[{\mathbf{y}}_a{\mathbf{x}}_b\right]$	$\mathbf{x}=\left[{\mathbf{x}}_a{\mathbf{y}}_b\right]$
$1\times 1$ conv [29]	${\mathbf{y}}_{:,i,j}=\mathbf{W}{\mathbf{x}}_{:,i,j}$	${\mathbf{x}}_{:,i,j}={\mathbf{W}}^{-1}{\mathbf{y}}_{:,i,j}$	$h.w.log|det\mathbf{W}|$
Our Shift Function	${\mathbf{y}}_{c,i,j}={\alpha }_c{\mathbf{x}}_{c,i,j}+{\beta }_c$	${\mathbf{x}}_{c,i,j}=[{\mathbf{y}}_{c,i,j}-{\beta }_c]/{\alpha }_c$	$h.w.{\sum }_{c}log\left|{\alpha }_c\right|$

. In particular, all functions easily obtain the reverse function and tractability of a Jacobian determinant. The symbols $\odot ,/$ denote element-wise multiplication and division. $h,w$ denotes the height and width of the input/output. $c,i,j$ are the depth channel index and spatial indices, respectively.

Coupling Layers: NICE [31] and RealNVP [32] presented coupling layers with a normalizing flow by stacking a sequence of invertible bijective transformation functions. The bijective function is designed as an affine coupling layer, which is a tractable form of Jacobian determinant. RealNVP can work in a multi-scale architecture to build a more efficient model for large inputs. To further improve the propagation step, the authors applied batch normalization and weight normalization during training. Later, Ho et. al. [43] presented a continuous mixture cumulative distribution function to improve the density modeling of coupling layers. In addition to improving the expressiveness of transformations of coupling layers [43], utilized multi-head self-attention layers [44] in the transformations.

Inverse Autoregressive Convolution: Germain et al. [45] introduced autoregressive autoencoders by constructing an extension of a non-variational autoencoder that can estimate distributions and is straightforward in computing their Jacobian determinant. Masked autoregressive flow [36] is a type of normalizing flow, where the transformation layer is built as an autoregressive neural network. Inverse autoregressive flow [30] formulates the conditional probability of the target variable as an autoregressive model.

Invertible $1\times 1$ Convolution: Kingma et al. [29] proposed simplifying the architecture via invertible $1\times 1$ convolutions. Learning a permutation matrix is a discrete optimization that is not amenable to gradient ascent. However, the permutation operation is simply a special case of a linear transformation with a square matrix. We can pursue this work with convolutional neural networks, as permuting the channels is equivalent to a $1\times 1$ convolution operation with an equal number of input and output channels. Therefore, the authors replace the fixed permutation with learned $1\times 1$ convolution operations.

Activation Normalization: [29] performs an affine transformation using scale and bias parameters per channel. This layer simply shifts and scales the activations with data-dependent initialization that normalizes the activations given an initial minibatch of data. This allows the scaling down of the minibatch size to 1 (for large images) and the scaling up of the size of the model.

Invertible $\mathit{n}\times \mathit{n}$ Convolution: Since the invertible $1\times 1$ convolution is not flexible, Hoogeboom et al. [35] proposed an invertible $n\times n$ convolution generalized from the $1\times 1$ convolutions. The authors presented two methods to produce the invertible convolutions: (1) Emerging Convolution and (2) Invertible Periodic Convolutions. Emerging Convolution is obtained by chaining specific invertible autoregressive convolutions [30] and speeding up this layer through the use of an accelerated parallel inversion module implemented in Cython. Invertible Periodic Convolutions transform data to the frequency domain via Fourier transform; this alternative convolution has a tractable Jacobian determinant and inverse. However, these invertible $n\times n$ convolutions require more parameters; therefore, these have an additional computational cost compared to our proposed method.

Lipschitz Constant: Behrmann et al. [37] developed a theory that any residual blocks satisfying the Lipschitz Constant can be invertible. Hence, Behrmann et al. proposed an invertible residual network (i-ResNet) as a normalizing flow-based model. Similar to [29,31,32,35], i-ResNet is learned by optimizing the negative log-likelihood in which the inverse flow and Jacobian determinant of the residual block can be efficiently approximated by the stochastic methods. Inheriting the success of Lipschitz theory, Kim et al. [38] proposed an $L_2$ self-attention that allows the self-attention of the Transformer networks [44] to be invertible.

3. Background

3.1. Flow-Based Generative Model

Let $\mathbf{x}$ be a high-dimensional vector with unknown true distribution $\mathbf{x}\sim p_{\mathcal{X}}\left(\mathbf{x}\right)$, $x\in \mathcal{X}$, a simple prior probability distribution $p_{\mathcal{Z}}$ on a latent variable $z\in \mathcal{Z}$, a bijection $f:\mathcal{X}\to \mathcal{Z}$, the change of variable formula defines a model distribution on $\mathcal{X}$ as shown in Equation (1).

$$p_{\mathcal{X}}\left(\mathbf{x}\right)=p_{\mathcal{Z}}\left(\mathbf{z}\right)|det\left(\frac{\partial f\left(\mathbf{x}\right)}{\partial \mathbf{x}}\right)|,$$

(1)

where $\frac{\partial f\left(x\right)}{\partial x}$ is the Jacobian of f at $\mathbf{x}$. The log-likelihood objective is then equivalent to minimizing:

$$\begin{array}{cc}\hfill \mathcal{L}\left(\mathcal{X}\right)& ={-}_{\mathbf{x}\in \mathcal{X}}log{p}_{\mathcal{X}}\left(\mathbf{x}\right)\hfill \\ \hfill & ={-}_{\mathbf{x}\in \mathcal{X}}\left[log{p}_{\mathcal{Z}}\left(\mathbf{z}\right)+log|det\left(\frac{\partial f\left(\mathbf{x}\right)}{\partial \mathbf{x}}\right)|\right].\hfill \end{array}$$

(2)

Since the data $\mathbf{x}$ are discrete data, we add a random uniform noise $u\in \mathcal{U}(0,a)$, where a is determined by the discretization level of the data, to make $\mathbf{x}$ be continuous data. The generative process can be defined as Equation (3).

$$\begin{array}{cc}\hfill \mathbf{z}& \sim p_{\mathcal{Z}}\left(\mathbf{z}\right)\hfill \\ \hfill \mathbf{x}& =f^{-1}\left(\mathbf{z}\right).\hfill \end{array}$$

(3)

The bijection function f is constructed from a sequence of invertible and tractable Jacobian determinant transformations: $f=f_1\circ f_2\circ \dots \circ f_K$ (K is the number of transformations). Such a sequence of invertible transformations is also called a normalizing flow. Here, Equation (2) can be written as in Equation (4).

$$\begin{array}{cc}\hfill \mathcal{L}\left(\mathcal{X}\right)& ={-}_{\mathbf{x}\in \mathcal{X}}log{p}_{\mathcal{X}}\left(\mathbf{x}\right)\hfill \\ & ={-}_{\mathbf{x}\in \mathcal{X}}\left[log{p}_{\mathcal{Z}}\left(\mathbf{z}\right)+\sum _{k=1}^{K}log|det\left(\frac{\partial {\mathbf{h}}_k}{\partial {\mathbf{h}}_{k-1}}\right)|\right]\hfill \end{array}$$

(4)

where ${\mathbf{h}}_k=f_1\circ f_2\circ \dots \circ f_k\left({\mathbf{h}}_0\right)$ with ${\mathbf{h}}_0=\mathbf{x}$.

3.2. Standard $n\times n$ Convolution

In this section, we revisit the standard $n\times n$ convolution. Let $\mathbf{X}$ be an $C\times H\times W$ input; $\mathbf{W}$ is a $D\times C\times K$ kernel, and the convolution can be expressed as follows:

$$\begin{array}{cc}\hfill \mathbf{Y}& =\mathbf{W}🟉\mathbf{X}=\left[{\mathbf{W}}_{:,:,1}{\mathbf{W}}_{:,:,2}\cdots {\mathbf{W}}_{:,:,k}\right]\times \left[\begin{array}{c}{\mathbf{X}}_{:,:,:}^1\\ {\mathbf{X}}_{:,:,:}^2\\ ⋮\\ {\mathbf{X}}_{:,:,:}^K\end{array}\right]\hfill \\ \hfill & =\sum _{k=1}^K{\mathbf{W}}_{:,:,k}\times {\mathbf{X}}_{:,:,:}^k=\sum _{k=1}^K{\mathbf{W}}_{:,:,k}\times {\mathcal{S}}_k\left(\mathbf{X}\right),\hfill \end{array}$$

(5)

where ${\mathbf{X}}_{:,:,:}^k$ is a $C\times H\times W$ matrix that represents a spatially shifted version of input matrix $\mathbf{X}$ with shift amount $(i_k,j_k)$,. ${\mathbf{W}}_{:,:,k}$ represents the $D\times C$ matrix corresponding to the kernel index k, the symbol 🟉 denotes a convolution operator.

In Equation (5), the standard convolution is simply a sum of $1\times 1$ convolutions on shifted inputs. The function ${\mathcal{S}}_k$ maps the input $\mathbf{X}$ to the corresponding shifted input ${\mathbf{X}}_{:,:,:}^k$. The standard convolution uses the common shifted input with integer-valued shift amounts for index k. Figure 2 illustrates our reformulated $n\times n$ convolution, if we can share the shifted inputs regardless of the kernel index, especially ${\mathcal{S}}_k\left(\mathbf{X}\right)=\mathcal{S}\left(\mathbf{X}\right)$, the standard convolution will be simplified as the $1\times 1$ convolution as shown in Equation (6). In this paper, we propose a shift function $\mathcal{S}$, which is an invertible and tractable form of the Jacobian determinant.

$$\begin{array}{cc}\hfill \sum _{k=1}^K{\mathbf{W}}_{:,:,k}\times {\mathcal{S}}_k\left(\mathbf{X}\right)& =\sum _{k=1}^K{\mathbf{W}}_{:,:,k}\times \mathcal{S}\left(\mathbf{X}\right)\hfill \\ \hfill & =\left(\sum _{k=1}^K{\mathbf{W}}_{:,:,k}\right)\times \mathcal{S}\left(\mathbf{X}\right).\hfill \end{array}$$

(6)

4. Invertible $\mathbf{n}\times \mathbf{n}$ Convolution

In this section, we first introduce our proposed Invertible Shift Function and then present invertible $n\times n$ convolution in details.

4.1. Invertible Shift Function

The shift function $\mathcal{S}$ will approximate all shifted input ${\mathbf{X}}_{:,:,:}^k$ ($1\le k\le K$). Here, we propose to design $\mathcal{S}$ as a linear transformation per channel; specifically, we have learnable variables ${\alpha }_c,{\beta }_c$; $1\le c\le C$ are scale and translation parameters for each channel, respectively. The shift function $\mathcal{S}$ can be formulated as follows:

$$\mathcal{S}\left({\mathbf{X}}_{c,i,j}\right)={\alpha }_c{\mathbf{X}}_{c,i,j}+{\beta }_c,$$

(7)

where $c,i,j$ are the depth channel index and spatial indices, respectively. The reverse function of $\mathcal{S}$ can be easy to obtain:

$${\mathbf{X}}_{c,i,j}=\frac{\mathcal{S}\left({\mathbf{X}}_{c,i,j}\right)-{\beta }_c}{{\alpha }_c}.$$

(8)

Thanks to Equation (7), the value of $\mathcal{S}\left({\mathbf{X}}_{c,i,j}\right)$ only depends on ${\mathbf{X}}_{c,i,j}$ and the Jacobian matrix will be in the form of the diagonal matrix as follows:

$$\begin{array}{cc}\hfill \mathbf{J}=\frac{\partial \mathcal{S}\left(\mathbf{X}\right)}{\partial \mathbf{X}}& =\left[\begin{array}{cccc}\frac{\partial \mathcal{S}\left({\mathbf{X}}_{1,1,1}\right)}{\partial {\mathbf{X}}_{1,1,1}}& 0& \cdots & 0\\ 0& \frac{\partial \mathcal{S}\left({\mathbf{X}}_{1,1,2}\right)}{\partial {\mathbf{X}}_{1,1,2}}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & \frac{\partial \mathcal{S}\left({\mathbf{X}}_{C,H,W}\right)}{\partial {\mathbf{X}}_{C,H,W}}\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{cccc}{\alpha }_1& 0& \cdots & 0\\ 0& {\alpha }_1& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\alpha }_c.\end{array}\right]\hfill \end{array}$$

(9)

Therefore, the determinant of Equation (9) is the product of all elements in the diagonal of the matrix $\mathbf{J}$ as in Equation (10).

$$\begin{array}{cc}\hfill det\left(\frac{\partial \mathcal{S}\left(\mathbf{X}\right)}{\partial \mathbf{X}}\right)& =\prod _{c=1}^C{\alpha }_c^{H\times W}\hfill \\ \hfill log|det\left(\frac{\partial \mathcal{S}\left(\mathbf{X}\right)}{\partial \mathbf{X}}\right)|& =H\times W\times \sum _{c=1}^{C}log\left|{\alpha }_c\right|.\hfill \end{array}$$

(10)

4.2. Invertible $n\times n$ Convolution

Kingma [29] proposed invertible $1\times 1$ convolution as the smart way to learn the permutation matrix instead of the fixed permutation [31,32]. However, the $1\times 1$ suffers from limited flexibility compared to the standard convolution. In particular, the receptive fields of $1\times 1$ convolution is limited. When the network goes deeper, the receptive fields of $1\times 1$ convolutions are still small areas; these, therefore, cannot generalize or model large objects of high-dimensional data. However, the $1\times 1$ convolution has its own advantages compared to the standard convolution. First, the $1\times 1$ convolution allows the network to compress the data of the input volume to be smaller. Second, $1\times 1$ suffers less over-fitting due to small kernel sizes. Therefore, in our proposal, we still take advantages of the $1\times 1$ convolution. Specifically, we adopt the successfully invertible $1\times 1$ convolution of Glow [29] in our design.

In the previous subsection, we proved that the shift function $\mathcal{S}$ is invertible and proved the tractability of the Jacobian determinant. In Section 3.2, we indicated that if we can share shifted inputs regardless of the kernel index via the shift function $\mathcal{S}$, we can simplify the standard $n\times n$ convolution to the composition of the $\mathcal{S}$ and $1\times 1$ convolution. Therefore, the invertible $n\times n$ convolution will be equivalent to the combination of the invertible shift function $\mathcal{S}$ and the invertible $1\times 1$ convolution. Specifically, the input will first be forwarded to the shift function $\mathcal{S}$ and then convoluted with the $1\times 1$ filter. Algorithm 1 illustrates the pseudo code of the invertible $n\times n$ convolution.

Algorithm 1: Invertible $n\times n$ Convolution

Input: An input $\mathbf{X}\in {\mathbb{R}}^{N\times H\times W\times C}$

Result: An output of invertible $n\times n$ convolution and the log Jacobian determinant

Initialize $\alpha ,\beta \in {\mathbb{R}}^C$ for the invertible shift function;

Initialize $\mathbf{W}\in {\mathbb{R}}^{C\times C}$ as a rotation matrix for the invertible $1\times 1$ convolution function;

logdet = 0.0;

The invertible shift function;

$\mathbf{Y}=\mathbf{X}\times \alpha +\beta$ (Channel-wise operations);

The inverse will be $\mathbf{X}=\frac{\mathbf{Y}-\beta }{\alpha }$;

logdet = logdet + ${\sum }_{i=1}^{C}log\left({\alpha }_i\right)$;

The invertible$1\times 1$convolution;

$\mathbf{Z}=Conv(\mathbf{Y},\mathbf{W})$;

The inverse will be $\mathbf{Y}=Conv(\mathbf{Z},{\mathbf{W}}^{-1}$);

logdet = logdet + $log(det(\mathbf{W}\left)\right)$ * H * W;

Return $\mathbf{Z}$ and logdet;

Figure 3a illustrates our one step of flow. We adopt the common design of a flow step [29,35,46] in our design. Our proposal can be easily integrated to the multi-scale architecture designed by Dinh et al. [32] (Figure 3b). From our proposal, we can generalize the invertible $1\times 1$ convolution to the invertible $n\times n$ convolution through the shift function $\mathcal{S}$. It can help to encourage the filters to learn a more efficient data representation and embed more useful latent features than the invertible $1\times 1$ convolution used in Glow [29]. Besides, we use fewer parameters and have less inference time compared to the standard $n\times n$ convolutions.

5. Experiments

In this section, we present our experimental results on CIFAR-10, ImageNet and Celeb-HQ datasets. Firstly, in Section 5.1, we compare log-likelihood against the previous flow-based models, that is, RealNVP [32], Glow [29] and Emerging Convolution [35]. Finally, in Section 5.2, we show our qualitative results trained on the Celeb-HQ dataset.

5.1. Quantitative Experiments

Datasets and Metric: We evaluate our invertible $n\times n$ convolution on CIFAR-10 (Figure 4a) and ImageNet (Figure 4b) with $32\times 32$ and $64\times 64$ image sizes. We use bits per dimension as the criteria with which to evaluate models. We compare our method against RealNVP [32], Glow [29] and Emerging Convolution [35]. We adopt the network structures of Glow and replace all invertible $1\times 1$ convolutions of Glow with our invertible $n\times n$ convolutions. For the data preprocessing, we follow the same process as in RealNVP [32].

Network Configurations: In the CIFAR experiment, the depth of flow K and the number of levels L are set to 32 and 3, respectively. Meanwhile, the depth of flow in ImageNet experiments is set to 48, the numbers of levels of ImageNet $32\times 32$ and ImageNet $64\times 64$ experiments are set to 3 and 4, respectively. We use the Adam optimizer [47] to optimize the networks in which batch size and learning rate are set to 64 (per GPU) and $0.001$, respectively. We choose Normal Distribution as the prior distribution $p_{\mathcal{Z}}\left(z\right)\sim \mathcal{N}(\mathbf{z};0,\mathbf{I})$ in all experiments.

The shift function $\mathcal{S}$ will be not inverse if the ${\alpha }_c=0$ ($\exists c\in [1\dots C]$). Hence, in the training process, we will first initialize ${\alpha }_c=1\mathrm{and}{\beta }_c=0$ ($1\le c\le C$). During the learning processing, we keep ${\alpha }_c$ ($1\le c\le C$) as a different 0 to guarantee that the shift function $\mathcal{S}$ is inverse and to guarantee the tractability of the Jacobian determinant. Training models on high-dimensional data requires large memory. To be able to train with a large batch size, we simultaneously and distributively trained the models on four GPUs via Horovod (https://github.com/horovod/horovod, accessed on 8 July 2021) and TensorFlow (https://tensorflow.org, accessed on 8 July 2021) frameworks.

Results:

Table 2. Comparative results (bits per dimension) of proposed invertible $n\times n$ convolution compared to RealNVP, Glow and Emerging Convolution.

Models	CIFAR-10	ImageNet 32	ImageNet 64
RealNVP	3.49	4.28	3.98
Glow	3.35	4.09	3.81
Emerging Conv	3.34	4.09	3.81
Ours	3.50	3.96	3.74

shows our experimental results. In particular, our proposal helps to improve the generative models on ImageNet $32\times 32$ and ImageNet $64\times 64$ datasets, which are more challenging than CIFAR-10. In particular, our proposed method achieves a state-of-the-art performance in which the bit per dimension results in ImageNet $32\times 32$, and ImageNet $64\times 64$ is 3.96 and 3.74, respectively. In comparison, the Emerging Convolution [35] and Glow achieve similar results in both ImageNet $32\times 32$ and ImageNet $64\times 64$ benchmarks, which are 4.09 and 3.81, respectively. Meanwhile, the corresponding results of RealNVP on these benchmarks are 4.28 and 3.98, respectively. As shown by the results, our proposed invertible $n\times n$ convolution provides a better generative capability than the stand-alone invertible $1\times 1$ convolution. Since Emerging Convolution uses invertible auto-regressive convolution, our proposal is, therefore, less complicated and has faster inference than Emerging Convolution. In the CIFAR-10 benchmark, although our model does not perform as well as Glow [29] and Emerging Convolution [35], we find it interesting that our method gains competitive results with a small number of modifications. The gap in performance is partially caused by the small amount of CIFAR-10 data that is inefficient for training the well-generalized convolution.

5.2. Qualitative Experiments

The CelebA-HQ dataset [42] was selected to train the model using the architectures defined in the previous section with a higher resolution ($256\times 256$ image sizes). The depth of flow K and the number of levels L were set to 32 and 6, respectively. Since high-dimensional data requires large memory, we reduced the batch size to 1 (per GPU) and trained on eight GPUs. The qualitative experiment aims to study the efficiency of the model when it scales up to the high-resolution images, synthesizes realistic images, and provides the meaningful latent space. Figure 4c shows the examples of Celeb-HQ datasets. We trained our model on 5-bit images in order to improve visual quality with a slight trade-off of color fidelity. As shown by the synthetic images in Figure 5, our model can generalize realistic images in high dimensional data.

6. Conclusions and Future Work

This paper has presented a novel invertible $n\times n$ convolution approach. By reformulating the convolution layer, we propose to use the shift function to shift inputs instead of kernels. We prove that our shift function is invertible and tractable in terms of calculating the Jacobian determinant. The method leverages the shift function and the invertible $1\times 1$ convolution to generalize to the invertible $n\times n$ convolution. Through experiments, our proposal has achieved state-of-the-art results in quantitative measurement and is able to generate realistic images with high-resolution.

There are several challenges that remain to be addressed in future work. In particular, when the model scales up to the high-resolution images, it requires a large amount of GPU memory during the training process, that is, the back-propagation process. Maintaining the rotation matrix property for the invertible $1\times 1$ convolution when training the model on a large dataset is also a challenging task, since the model easily falls into the non-inverse matrix due to the stochastic gradient update of the back-propagation algorithm. That issue is interesting work and should be improved in the future.