Non-Intrusive Inference Reduced Order Model for Fluids Using Deep Multistep Neural Network

Abstract

In this effort we propose a data-driven learning framework for reduced order modeling of fluid dynamics. Designing accurate and efficient reduced order models for nonlinear fluid dynamic problems is challenging for many practical engineering applications. Classical projection-based model reduction methods generate reduced systems by projecting full-order differential operators into low-dimensional subspaces. However, these techniques usually lead to severe instabilities in the presence of highly nonlinear dynamics, which dramatically deteriorates the accuracy of the reduced-order models. In contrast, our new framework exploits linear multistep networks, based on implicit Adams–Moulton schemes, to construct the reduced system. The advantage is that the method optimally approximates the full order model in the low-dimensional space with a given supervised learning task. Moreover, our approach is non-intrusive, such that it can be applied to other complex nonlinear dynamical systems with sophisticated legacy codes. We demonstrate the performance of our method through the numerical simulation of a two-dimensional flow past a circular cylinder with Reynolds number Re = 100. The results reveal that the new data-driven model is significantly more accurate than standard projection-based approaches.

1. Introduction

The full order model (FOM) of realistic engineering applications in fluid dynamics often represents a large scale dynamic system. High-fidelity computational fluid dynamics (CFD) simulations of the FOM are so computationally expensive that they put a heavy burden on the computational resources despite the available CFD software and supercomputers with thousands of cores. Consequently, the use of FOM for such simulations is often impractical and prohibitive for time-critical applications such as system identification, flow control, design optimization.

The reduced order modeling in fluid dynamics is used to construct an accurate low-dimensional approximation to the full system with orders of magnitude reduction in computational cost. The first usage of reduced order model (ROM) in fluid dynamics was by [1] for studying the intensity of turbulence and coherent structures. Many recent successful applications of ROM in fluid problems can be found in [2,3,4,5,6,7,8,9].

The Galerkin projection based reduced order model (GP-ROM) is one of the most popular methods that has been widely used in practice. The GP-ROM, in an offline stage, first constructs a reduced space and then uses the Galerkin projection of the FOM operator to obtain a low-dimensional nonlinear dynamical system, i.e., ROM dynamics. The reduced space is often generated by proper orthogonal decomposition (POD), also known as principal component analysis. In an online stage, the obtained reduced dynamics can be used to approximate the full system efficiently for various applications, such as long-term prediction, flow control. However, the projection step requires that the full model operators have to be available in order to obtain the ROM dynamics. This limits the applicability of projection based model reduction in situations where the full model is unknown [10,11]. More importantly, the computational cost of assembling the reduced operators—tensors from the projection of FOM operator—scales with the large dimension of the underlying high-dimensional of FOM. For this reason, the GP-ROM are efficient for problems where the reduced operators must be constructed only once.

On the other hand, the GP-ROM generates inaccurate approximations for highly non-stationary (nonlinear) fluids, e.g., turbulence. In the literature, a common explanation for this failure is that the Galerkin projection does not preserve the stability properties from the full model. A deeper reason is that the low-dimensional space used in the Galerkin projection cannot resolve the nonlinear interaction of the fluid system [12,13,14]. Consequently, resulting in a projection based stability error which makes GP-ROM fail in nonlinear fluid applications. Balajewicz [15] studied the application of low dimensional modelling of shear flows; Ballarin [16] applied the POD-Galerin to the parametrized steady incompressible Navier–Stokes equations (NSE); Xie [17] introduced filtered ROM method for 2D cylinder flow.

1.1. Related Work

Closure modeling. Numerous stabilization strategies have been devised to address the instability problem, known as closure modeling. The fundamental idea of closure modeling is to model the lost information from the low-dimensional space since it is generated through POD truncation. This truncation is to keep the first few POD modes that extract the most dominant structure of the full system and discard the rest modes. Closure models generally can be categorized into two common approaches. One physically models the effect of the discarded POD mode by adding artificial viscosity to the reduced system [6,15,16,18]. The other to mathematically models the ROM dynamics by solving a related optimization problem [14,17]. Another data-driven approach is the POD with a trained radial basis function (RBF) network to estimate various physical paramters with little prior knowledge of the system. The POD-RBF achieves good numerical robustness and stability, see [19,20,21]. The most recent development for closure modeling is to apply the neural network to approximate the lost information [22].

Differential equations learning. Bridging numerical differential equations and deep neural networks have gained enormous attention recently. Chang [23] proposed dynamical system viewing of residual networks (ResNet). Lu [24] introduced the linear multistep network architecture for the first time to analyze ResNet on classification task. Sparse regression, Gaussian process, multistep neural networks have been applied for the data-driven discovery of dynamic systems [25,26,27,28,29]. More recently, an ordinary differential equation network (ODE-net) was introduced for supervised learning [30]. In this work, we focus on the traditional projection based reduced order modeling process and make improvement by using the state-of-the-art deep learning method.

1.2. Our Approach

In this paper, we propose a novel non-intrusive learning reduced order model framework for a fluid dynamic system. The new framework provides a general concept of learning the optimal reduced dynamic system from the data. Inspired by the successful development in learning differential equations with deep networks, we apply the linear multistep neural network (LMNet) to learn the reduced order model (LMNet-ROM). Unlike closure modeling and the existing non-intrusive model reduction methods, in this work, we focus on a different perspective. First, we do not use Galerkin projection and model the closure problem. The optimal reduced dynamic system that can address the instability issue is learned for a given supervised learning task. Secondly, the new model does not approximate reduced operators whereas other non-intrusive models use interpolation or regression method to infer reduced operators [10,11]. Moreover, our viewpoint easily enables us to answer the common question—what is the best ROM dynamic system to approximate the full system for a given low-dimensional subspace? We demonstrate the performance of the new LMNet-ROM is better than GP-ROM in full order model approximation and long-term prediction. The main contributions can be summarized as follows:

• A novel non-intrusive learning reduced order model framework for fluid dynamics, which is applicable to general nonlinear dynamical systems with sophisticated legacy codes.
• Our framework overcomes the instability issue of the projection based model reduction, and provides accurate approximation and long-term prediction of the full system.
• The learning process of our approach is more computationally efficient than the construction of reduced operators in the classic projection based methods.

2. Reduced Order Modeling

In this section, we present the Galerkin projection based reduced order modeling framework in fluid dynamic system. The classical Navier–Stokes equations (NSE) are often used as a mathematical model in fluid dynamics,

$$\frac{\partial \mathit{u}}{\partial t}-R{e}^{-1}\Delta \mathit{u}+\mathit{u}·\nabla \mathit{u}+\nabla p=\mathbf{0},$$

(1)

$$\nabla ·\mathit{u}=0,$$

(2)

where $\mathit{u}$ is the velocity, p the pressure, and $Re$ the Reynolds number. We use the initial condition $\mathit{u}(\mathit{x},0)={\mathit{u}}_0\left(\mathit{x}\right)$ and (for simplicity) homogeneous Dirichlet boundary conditions: $\mathit{u}(\mathit{x},t)=\mathbf{0}$. For convenience, we use $\frac{\partial \mathit{u}}{\partial t}=\mathbf{f}(\mathit{u},Re,p)$ as the general notation for the NSE in the rest of the paper.

Reduced Space. Proper orthogonal decomposition (POD) is the dimension reduction method that we used to generate the reduced space. It starts with the data matrix, $\mathit{U}=[{\mathit{u}}^0,{\mathit{u}}^1,\cdots ,{\mathit{u}}^s]\in {\mathcal{R}}^{D_h\times (s+1)}$, collected by the numerical solutions or experimental observations of the full system (1) at $s+1$ different time instances. The POD method seeks a low-dimensional space ${\mathit{X}}^r$ that approximates the data ($\mathit{U}$) optimally with respect to the $L^2$-norm. It formulates the following eigenvalue problem:

$$\mathit{U}{\mathit{U}}^T{\phi }_i={\lambda }_i{\phi }_i,i=1,2,\dots ,d,$$

(3)

where d is the rank of the data matrix $\mathit{U}{\mathit{U}}^T$. ${\lambda }_i$ and ${\phi }_i$ are the eigenvalues and POD basis, respectively. The reduced space is given after the truncation as ${\mathsf{\Phi }}_r:=\{{\phi }_1,\cdots ,{\phi }_r\}\in {\mathcal{R}}^{D_h\times r}$.

Galerkin projection-ROM (GP-ROM). For a given space ${\mathsf{\Phi }}_r$, the GP-ROM finds the approximation of the velocity field spanned by the low-dimensional space,

$$\mathit{u}\approx {\mathit{u}}_r(\mathbf{x},t)\equiv \sum _{j=1}^r{a}_j\left(t\right){\mathit{\phi }}_j\left(\mathbf{x}\right),$$

(4)

where ${\left\{a_j\left(t\right)\right\}}_{j=1}^r$ are the sought time-varying coefficients. The GP-ROM can be obtained by projecting the FOM onto the POD space: $\forall i=1,\dots ,r,$

$$\left(\frac{\partial {\mathit{u}}_r}{\partial t},{\mathit{\phi }}_i\right)=\left(\mathbf{f}({\mathit{u}}_r,Re),{\mathit{\phi }}_i\right).$$

(5)

Here, $(·,·)$ is the $L^2$ inner product. Note that the POD basis functions are weakly divergence free, i.e., $\nabla ·{\mathit{\phi }}_i=0$, so the pressure term will not be included in the ROM projection Equation (5) as $(p,\nabla ·{\mathit{\phi }}_i)=0$. The solution of GP-ROM can be determined by the following nonlinear dynamic system:

$$\dot{\mathit{a}}=L\mathit{a}+{\mathit{a}}^{\top }N\mathit{a},$$

(6)

where L and N are ROM operators that can be obtained by projection

$$L_{im}=\frac{1}{Re}\left(\Delta {\mathit{\phi }}_m,{\mathit{\phi }}_i\right),N_{imn}=-\left({\mathit{\phi }}_m·\nabla {\mathit{\phi }}_n,{\mathit{\phi }}_i\right).$$

(7)

Note that the reduced system (6) efficiently approximates the full model of NSE as the dimension r is generally very small ($\sim \mathcal{O}\left(10\right)$) compared to the high dimension of $\mathit{u}$ ($D_h\sim \mathcal{O}(\ge {10}^5)$).

ROM closure models. The ROM closure models can be generally written as the following dynamic system:

$$\dot{\mathit{a}}=L\mathit{a}+{\mathit{a}}^{\top }N\mathit{a}+\mathit{\tau },$$

(8)

where $\mathit{\tau }$ is an artificial term that model the effect of the discarded POD modes using various approaches e.g., see [17,22,31,32,33]. We note that the dynamics of the closure model (8) is more accurate than GP-ROM dynamics (6) in the approximation of the model, relevant studies can be found in [2,12,13,14]. The challenge of closure modeling is that the $\mathit{\tau }$ is unknown, i.e., no explicit formula. Therefore, closure models are empirical modification of the GP-ROM from physical or mathematical perspective.

3. Learning Reduced Order Model

In this section, we present the architecture of learning the reduced order model from deep neural networks. In contrast to the standard GP-ROM framework, we learn the reduced dynamical system from the data without intrusively using ROM operators (e.g., $L,N$)

3.1. Rom Dynamics

We consider the low dimensional ROM dynamic system as a general function,

$$\dot{\mathit{a}}={\mathbf{f}}_r\left(\mathit{a}\right).$$

(9)

We claim that this function ${\mathbf{f}}_r$ has a general representation of the ROM dynamics including system (6) and (8). Our goal is to learn the ROM dynamics (9) in a given set of temporal data and return a closed form model that can be used to accurately approximate and predict the full system.

For a given data-set of snapshot solutions of NSE (1), $\mathit{U}={\mathit{u}}^0,\dots ,{\mathit{u}}^s\in {\mathcal{R}}^{D_h\times s+1}$, at time steps $t_0,\dots ,t_s$. The best approximation of snapshots data by the POD space is given by, ${\mathit{u}}^j={\sum }_{i=1}^r{b}_i^j{\phi }_i$, $j=t_0,\dots ,t_s$. The reduced dynamics (9) is to find coefficients $\mathit{a}$ such that,

$${\mathit{u}}_r=\sum _{i=1}^r{a}_i^j{\phi }_i\approx \mathit{u}=\sum _{i=1}^r{b}_i^j{\phi }_i.$$

(10)

It indicates that the optimal solution from the r-dimensional ROM dynamic system is given by the full model data, ${\mathit{b}}^j$, such that

$$\dot{\mathit{b}}={\mathbf{f}}_r\left(\mathit{b}\right).$$

(11)

This provides a framework in which the ROM dynamics can be learned from the data-set $B=[{\mathit{b}}^0,\dots ,{\mathit{b}}^s]\in {\mathcal{R}}^{r\times (s+1)}$, i.e., time-varying coefficients of FOM data. The training data-set can be computed by the following,

$$B={\mathsf{\Phi }}_r^{T}W\mathit{U}.$$

(12)

The formula is derived by the $L^2$ projection of data $\mathit{U}$ to the low-dimensional space ${\mathsf{\Phi }}_r$. $W\in {\mathcal{R}}^{D_h\times D_h}$ is the weight matrix of the $L^2$ inner product, we use finite element weights in this work.

3.2. Linear Multistep Network (LMNet)

Motivated by differential equation learning, we adopt the linear multistep network architecture [24,34] to construct a structured nonlinear regression model that can learn the reduced dynamics. In this work, we only consider the implicit multistep method, Adams–Moulton (AM) scheme [35] as it has better stability property. The K-step AM method is defined as follows:

$${\mathit{a}}^n=\sum _{i=1}^K({\alpha }_i{\mathit{a}}^{n-i}+{\beta }_i\Delta t{\mathbf{f}}_r\left({\mathit{a}}^{n-i}\right))+{\beta }_0\Delta t{\mathbf{f}}_r\left({\mathit{a}}^n\right)$$

(13)

We discretize the ROM dynamic system (9) by using the AM scheme (13) with a neural network. The parameters of this neural network can be learned by minimizing the mean squared error loss function:

$$MSE:=\frac{1}{N-K+1}\sum _{n=K}^N{\left|{\mathcal{L}}_n\right|}^2,$$

(14)

where N is the total number of time instance in the ROM dynamic system, K denotes the step for AM method. ${\mathcal{L}}_n$ is the local truncation error from the Taylor expansion of K-step AM method (13),

$${\mathcal{L}}_n=\sum _{i=0}^K{\alpha }_i{\mathit{a}}^{n-i}-\Delta t{\beta }_i{\mathbf{f}}_r\left({\mathit{a}}^{n-i}\right).$$

(15)

The derivation of the loss function follows the truncation error of the multistep method which can be obtained via a Taylor series, and we add the formula in the Appendix A, while other references can be found in [35,36]. The goal of this neural network was to approximate the function ${\mathbf{f}}_r$ with the input of the network is the coefficients $\mathit{a}$ and the output of the network is the ${\mathbf{f}}_r\left(\mathit{a}\right)$ representing the time derivative. This neural network architecture uses a family of implicit linear mulistep method to discretize the ROM dynamic system (9). The implicit scheme comes with the last term ${\beta }_0\Delta t{\mathbf{f}}_r\left({\mathit{a}}^n\right)$ such that ${\beta }_0\ne 0$. In neural network training, the parameters ${\alpha }_i,{\beta }_i$ are given for a certain K-step AM method. Also, we do not need to compute the implicit term since the coefficients ${\mathit{a}}^n$ is known (i.e., the training data), which makes the implementation simple. The target function will be trained by minimizing the cost function (15). Another advantage of this nonlinear regression is that we do not have to approximate the temporal gradients [11,26] since the time derivatives are discretized by the AM method. Our approach is different from the method in PolyNet [37]. The PolyNet focused on the inception of residual neural network block and can be interpreted as an approximation to one step of the backward Euler scheme, i.e., ${\mathit{a}}^n={(I-\Delta t\mathbf{f})}^{-1}{\mathit{a}}^{n-1}=[I+\Delta t\mathbf{f}+{(\Delta t\mathbf{f})}^2+\cdots +{(\Delta t\mathbf{f})}^n+\cdots ]{\mathit{a}}^{n-1}$, I denotes the identity mapping. The author used second order approximation in their paper [37].

3.3. LMNet-ROM

We use the trained neural network as the ROM dynamic system (9) to approximate the full system of NSE. We emphasize that the novelty of our approach is the non-intrusively learning of the reduced system, whereas GP-ROM and closure models require the use of FOM operators. Figure 1 is the flowchart of the LMNet-ROM and GP-ROM framework.

We outline the Algorithm 1 of the framework in the following:

Algorithm 1 Linear multistep network reduced order model learning (LMNet-ROM).

Compute the reduced POD space from the data of NSE by (3)  Compute the training dataset B via (12)  Train the neural network using loss function (14)  The LMNet-ROM for NSE is obatined from the trained low-dimensional dynamic system: $$\dot{\mathit{a}}={\mathbf{f}}_r^{Net}\left(\mathit{a}\right)$$ (16)

We claim that the learned reduced dynamics (16) has a better approximation to the full model than system (7) and (8) since it is learned optimally from the FOM data. The new framework, see Figure 1, only requires input data from a system and does not use any FOM operator, which can be generally applied to reduced order modeling of any fluid dynamical system. The main offline computational cost of the LMNet-ROM is training the neural network whereas the GP-ROM is the construction of ROM operators in (7). In the numerical experiment, we show that the offline computation of our model is faster than the GP-ROM.

4. Numerical Experiment

In this section, we present the preliminary numerical results to demonstrate the advantages of our new model. The test case is a 2D channel flow past a circular cylinder at a $Re=100$. It is a benchmark problem that has been used as a numerical test in fluid dynamics, see [8,17,38,39,40].

4.1. Implementation Details

The domain is a $2.2\times 0.41$ rectangular channel with a radius=$0.05$ cylinder, centered at $(0.2,0.2)$, see Figure 2. No slip boundary conditions are prescribed for the walls and on the cylinder, and the inflow and outflow profiles are given by [17,40] $u_1(0,y,t)=u_1(2.2,y,t)=\frac{6}{{0.41}^2}y(0.41-y),u_2(0,y,t)=u_2(2.2,y,t)=0$. The kinematic viscosity is $\nu ={10}^{-3}$, there is no forcing, and the flow starts from rest.

The velocity snapshots of NSE (1) were generated by a finite element method with meshsize approximately 103,000 which gives a fully resolved solution. The lift and drag computation agree well with results from references in [41]: $c_{d,max}=3.2261,c_{l,max}=1.0040$.

A total number of 2500 snapshots $\mathit{U}$ were collected from $T=[0,5]$ at every time step $\Delta t=0.002$. The LMNet-ROM was built and trained in Tensorflow.

4.2. Full Order Model Approximation

After obtaining the trained neural network, we used the ROM dynamic system (16) to approximate the full model of NSE (1). The following average $L^2$ error formula has been used to quantify the accuracy of the model,

$$\mathcal{E}=\frac{1}{s+1}\sqrt{\sum _{j=0}^s{\int }_{\mathsf{\Omega }}({\mathit{u}}^j-{\mathit{u}}_r^j)d\mathsf{\Omega }}.$$

(17)

We first evaluated the model with different layers and neurons. The dimension of the ROM dynamics is fixed to be $r=8$ with the one step Adams–Moulton method.

Table 1. Average $L^2$ error between trajectories of the learned reduced order model with dimension $r=8$ and the exact data for the different number of hidden layers and neurons.

	Neurons	64	128	256
Layers
1		$5.62\times {10}^{-2}$	$7.22\times {10}^{-3}$	$4.31\times {10}^{-1}$
2		$4.01\times {10}^{-3}$	$5.75\times {10}^{-3}$	$2.95\times {10}^{-2}$
3		$1.64\times {10}^{-2}$	$4.04\times {10}^{-3}$	$3.11\times {10}^{-3}$

provides a crude estimate that increasing network width (256 neurons) might have the potential over-fitting issue whereas decreasing units (64) may not be enough to reach a good accuracy. Also, the network depth (number of hidden layers) have a positive effect on the performance of the model. To reduce the numerical efforts, we used one hidden layer and 128 neurons for the neural network in the rest of our evaluations. We emphasize that to fully understand the model sensitivity with respect to network architecture, a systematic study involving regularization, batch normalization and drop out is needed in future research.

We also test the LMNet-ROM with different steps in the training.

Table 2. Average $L^2$ error between the new model and exact data for the different number of steps and dimensions.

	Dimension	r = 4	r = 6	r = 8
K/model
1		$1.88\times {10}^{-3}$	$3.05\times {10}^{-3}$	$7.22\times {10}^{-3}$
2		$2.83\times {10}^{-4}$	$6.04\times {10}^{-4}$	$1.45\times {10}^{-3}$
3		$2.67\times {10}^{-4}$	$6.22\times {10}^{-4}$	$7.14\times {10}^{-3}$
4		$3.79\times {10}^{-4}$	$6.20\times {10}^{-4}$	$6.37\times {10}^{-4}$
GP-ROM		$1.66\times {10}^{-1}$	$7.30\times {10}^{-2}$	$1.92\times {10}^{-2}$

lists the error between the new model and the exact data. As a comparison, we add GP-ROM result in the last entry. Table 2 shows that the LMNet-ROM consistently provide more accurate results as increasing steps of Adams–Moulton (AM) method. An intuitive explanation is that the stability property of AM method helps to regularize the network and eventually achieve a good calibration. Large steps (K) requires high computational cost of training the network. To balance the output accuracy and training cost, we used step $K=1$ for the rest of our numerical tests.

Noisy data. The above numerical tests are carried out on the deterministic data. In some situations, however, problems may contain noise measurements. We study the robustness of the new method with respect to noise data by adding Gaussian noise to the data-set for both models.

Table 3. Average $L^2$ error for different noise magnitudes.

	Model	GP-ROM	LMNet-ROM
Noise
0.0%		$1.92\times {10}^{-2}$	$7.22\times {10}^{-3}$
0.5%		$1.93\times {10}^{-2}$	$2.44\times {10}^{-3}$
1%		$2.05\times {10}^{-2}$	$2.20\times {10}^{-2}$
5%		$2.92\times {10}^{-2}$	$8.76\times 10$

lists the error comparison between the LMNet-ROM and GP-ROM for different level of noise data. The results show that the LMNet-ROM cannot preserve good performance when the noise magnitude (≥1%) is high while GP-ROM does. The argument is that the learned dynamical system fully depends on the data which makes it vulnerable to noise interference. The GP-ROM, however, requires the use of FOM operators making it less sensitive than LMNet-ROM to noise. Further approaches should be investigated for the potential improvement for this problem. As for deterministic data, the LMNet-ROM is better than GP-ROM in full system approximation.

4.3. Long-Term Prediction

In this section, we make a thorough study of long-term predictability of the new LMNet-ROM. The solution to our new model and the GP-ROM were computed by the dynamic system (16) and (6), respectively. We used direct numerical simulation (DNS) to denote the exact solution (data) of NSE. We used snapshot data that were collected from the time interval $T=[0,3]$ to generate the POD space and train the neural network. We then run the reduced systems (16) and (6) for $T=[0,5]$ to make the prediction. Figure 3 plots the phase portraits of the first few coefficients, $a_2,a_3,a_4$, from both models and the DNS data. The red line that depicts the result from LMNet-ROM has a closer mimic of DNS data, whereas the portraits from the GP-ROM have small deviation. This behavior tells that the dynamics of LMNet-ROM predict future states better than GP-ROM with the given information. Note that $a_1$ was a constant and not meaningful to discuss since the first POD mode ${\phi }_1$ represents the mean flow.

We also looked at the prediction of the time evolution of energy ($E\left(t_j\right)=\frac{1}{2}{\parallel {\mathit{u}}^j\parallel }_{L^2}$), vorticity construction, and drag coefficients. Figure 4 shows the long-term prediction of energy travel and drag. The main observation is that the LMNet-ROM performs much better than GP-ROM. The energy and drag generated from GP-ROM have a huge deterioration when time evolves, which means the prediction is not accurate. Clearly, the prediction from LMNet-ROM is impressively good as it is stable and close to DNS, see Figure 4. Vorticity describes the local spinning motion of a fluid system. Figure 5 plots the vorticity construction from the velocity field around the cylinder at the end time $T=5$. As depicted in Figure 5, the LMNet-ROM correctly predicts the vortex street behind the cylinder while GP-ROM not. The above results are presented for dimension $r=8$, but similar results can be found for $r=4,6$. Overall, the long-term predictability of LMNet-ROM is much better than GP-ROM.

4.4. LMNet-ROM vs. Closure Models

In this section, we present a preliminary numerical comparison between the LMNet-ROM and the closure models. Due to the wide class of stabilization closure models and the availability of their open source implementation, a thorough comparison between our new model and all other models is not practical. Therefore, we use the two most recent models (open source implemenation), data-driven-filtered ROM (DDF-ROM) and evolve-then-filter ROM (EF-ROM), to illustrate the numerical comparison. The EF-ROM uses a two steps regularization strategy to improve the GP-ROM [33]. The first step was to use the forward Euler discretization to evolve the GP-ROM dynamic system (7). Then it followed a ROM filter to smooth the solution for regularization purpose, more details can be found in [33]. The DDF-ROM solveds an optimization problem to approximate the $\mathit{\tau }$ in system (8) [17].

Table 4. Average $L^2$ error from different models.

	Dimension	r = 4	r = 6	r = 8
Model
EF-ROM		$1.23\times {10}^{-1}$	$7.31\times {10}^{-2}$	$1.84\times {10}^{-2}$
DDF-ROM		$2.27\times {10}^{-1}$	$1.14\times {10}^{-2}$	$1.22\times {10}^{-2}$
LMNet-ROM		$1.91\times {10}^{-3}$	$3.57\times {10}^{-3}$	$7.22\times {10}^{-3}$

lists the average $L^2$ error from the three models with different dimensions. It surprisingly indicates that the closure models with mathematical methods cannot outperform the LMNet-ROM. This result was impressive, given that the LMNet-ROM is learned through the neural network from the data without acquiring any FOM information.

We note that the goal of both the stabilization closure models and our new model was to provide an accurate reduced dynamics to approximate the full system from different perspective. The former generally model the artificial term ($\mathit{\tau }$) in the system (8) physically or mathematically, whereas the latter use data to learn the dynamics (16). We claim that the non-intrusive learning framework has the generality that can be applied to any nonlinear fluid system since it only requires the training data.

4.5. Computational Cost

In this section, we discuss the computational efficiency of the proposed LMNet-ROM. The main computational cost of reduced order models is the offline computation since the cost of solving a small ODE system is negligible at the online stage. The FOM simulation (DNS) time is used as a benchmark to evaluate the performance of each model. The computation was carried out on a 64-bit Linux system with a single 2.70 GHz CPU. The DNS CPU time is 36,828.53(s).

Table 5. Offline cost (second) and speed up factor from each reduced order model (ROM) with dimension $r=8$.

Model	Cost	Speed-Up Factor ($\frac{\mathit{DNS}}{\mathit{ROM}}$)
GP-ROM	855.52 s	43.05
EF-ROM	867.20 s	42.47
DDF-ROM	6373.97 s	5.78
LMNet-ROM	445.25 s	82.71

lists the CPU time from each ROM and the associated speed-up factor. The LMNet-ROM time is only computed from one step AM method with one layer and 128 neurons, given the fact that this network architecture achieves good accuracy in the previous test. The result in Table 5 reveals that the LMNet-ROM is more efficient compared to other models. This is a huge advantage of our new method since reducing the computational cost while maintaining good accuracy is the primary goal of reduced order modeling.

5. Conclusions and Outlook

In this paper, we proposed a novel learning reduced order model framework for the numerical simulation of fluid flows. This framework was based on the recent development of linear multistep network architecture. We numerically studied the LMNet-ROM in the simulation of a 2D flow past a cylinder. The numerical results demonstrate that the LMNet-ROM was significantly more accurate than the GP-ROM in system approximation and long-term prediction. Furthermore, we compared the new model with the two most recent stabilization closure models, EF-ROM and DDF-ROM. The results show that our new method outperforms the two closure models. Overall, the LMNet-ROM beats the aforementioned models both in accuracy and computational efficiency, which provides a promising and encouraging approach in model reduction of fluid dynamics. The main advantage of using our approach is the fully data-driven process while maintain good numerical accuracy. The model does not require the stored ROM operator whereas other ROM method does. However, the LMNet-ROM’s potential still needs to be explored. We outline some research directions that could be pursued.

Probably the most important next step is to study the parametrized system prediction of the LMNet-ROM. The current neural network is trained under the data-set given by one parameter value ($Re$) from the full order model of NSE. How does the LMNet-ROM predict systems with different parameter values, such as initial conditions and boundary conditions? Parametrized system prediction is a challenging problem in engineering applications. We hope to provide a systematic investigation with the new LMNet-ROM in later research.

Another important research direction is to improve the robustness of the model with respect to noise data. Table 3 shows the drawback of this model for high magnitude noisy (≥1%) data. We plan to address this issue by improving the neural network architecture. Also, regularization for preventing over-fitting needs to be fully studied in future research.

Finally, the generality study of LMNet-ROM is worth investigation. Although we constructed and tested the LMNet-ROM in a fluid dynamics setting, the LMNet-ROM framework can be applied to any type of nonlinear partial differential equations (PDE) that is amenable to reduced order modeling. The only input needed in the LMNet-ROM framework is the data from the FOM of any system, see Algorithm 1. The LMNet-ROM procedure does not restrict it to the particular physical system modeled by the given nonlinear PDE. Since the LMNet-ROM is built by fully data-driven learning, we expect it to be successful in the numerical simulation of general mathematical models (e.g., from elasticity or bioengineering).