A Comparison between Successive Estimate of TVAR(1) and TVAR(2) and the Estimate of a TVAR(3) Process ^†

Abstract

In time series analyses, the auto-regressive (AR) modelling of zero mean data is widely used for system identification, signal decorrelation, detection of outliers and forecasting. An AR process of order p is uniquely defined by p coefficients and the variance in the noise. The roots of the characteristic polynomial can be used as an alternative parametrization of the coefficients, which can be used to construct a continuous covariance function of the AR process or to verify that the AR process is stationary. In a previous study, we introduced an AR process of time variable coefficients (TVAR process) in which the movement of the roots was specified as a polynomial of order one. Until now, this method was analytically derived only for TVAR processes of orders one and two. Thus, higher-level processes had to be assembled by the successive estimation of these process orders. In this contribution, the analytical solution for a TVAR(3) process is derived and compared to the successive estimation using a TVAR(1) and TVAR(2) process. We will apply the proposed approach to a GNSS time series and compare the best-fit TVAR(3) process with the best-fit composition of TVAR(2) and TVAR(1) process.

1. Introduction

The auto-regressive process is a means in time series analysis and, among other things, this method is used to estimate discrete covariance functions (see [1] [p. 32, eq. (182)]) or to decorrelate observations by filtering [2,3,4]. Under the assumption of a non-stationary process, there are two possible states: first, that the roots of the characteristic polynomial are no longer in the unit circle, and therefore no covariances can be calculated, and secondly, the case we are looking at here, where the root changes over time, but always stays in the unit circle. This case has often appeared in the literature, but the TVAR coefficients, when used, were given a form of motion like polynomial motions (see [5,6]), trigonometric function [7], modified Legendre polynomials [8] and spherical sequences [9]. What all these methods often overlook is the resulting movement of the roots, as can be seen, for example, in [10]. In [11], it was derived how a TVAR process has to be calculated successively from TVAR(1) and TVAR(2) process estimates for a linear root movement. This method is now to be extended so that it can be applied directly to the TVAR(3) process without testing different series of TVAR(1) and TVAR(2) processes. Divided into five chapters, this extension is presented here. In Section 2, the necessary condition for the linear roots of a TVAR process of arbitrary order is derived again. In Section 3, additional restrictions are derived as sufficient conditions. In Section 4, the theory is tested on a GNSS time series and contrasted with successive estimations. The discussion of the results, as well as an overlook of further research directions, can be found in Section 5.

2. Successive Estimation Using TVAR(1) and TVAR(2) Processes

Ref. [10] defines the time variable auto-regressive process of the order p (TVAR (p) process) using the recursion formula

$$\begin{array}{c}\hfill {𝒮}_t={\alpha }_1\left(t\right){𝒮}_{t-1}+{\alpha }_2\left(t\right){𝒮}_{t-2}+...+{\alpha }_p\left(t\right){𝒮}_{t-p}+{\mathcal{E}}_t.\end{array}$$

(1)

Here ${\alpha }_1\left(t\right)$, ${\alpha }_2\left(t\right)$, …, ${\alpha }_p\left(t\right)$ are the time-varying coefficients of the TVAR(p) process, and ${\mathcal{E}}_t$ is an i.i.d. sequence with variance ${\sigma }_{\mathcal{E}}^2$. As long as a single time $t=\tau$ is considered separately, the representation in (1) corresponds to a time-stable AR process of order p (TSAR(p) process):

$$\begin{array}{c}\hfill {𝒮}_t={\alpha }_1\left(\tau \right){𝒮}_{t-1}+{\alpha }_2\left(\tau \right){𝒮}_{t-2}+...+{\alpha }_p\left(\tau \right){𝒮}_{t-p}+{\mathcal{E}}_t.\end{array}$$

According to [12] [p. 167], the TSAR Process is stationary if the roots ($P_1$, $P_2$, …, $P_p$) of the characteristic polynomial

$$\begin{array}{cc}\hfill \chi \left(x\right)& =x^p-{\alpha }_1\left(\tau \right)x^{p-1}-{\alpha }_2\left(\tau \right)x^{p-2}-...-{\alpha }_p\left(\tau \right)\hfill \\ \hfill & =(x-P_1\left(\tau \right))(x-P_2\left(\tau \right))...(x-P_p\left(\tau \right))\hfill \end{array}$$

are in the unit circle ($\parallel P_k\left(\tau \right)\parallel <1$∀$k=1,2,..,p$). Ref. [11] has shown how TVAR processes with linear root movements are successively estimated using TVAR(1) and TVAR(2) processes, and the advantages of this method compared to the estimation of time constant AR processes. The general TVAR estimate consists of two steps: First, the coefficients of the TVAR process are replaced by polynomials, where the order of the polynomial is equal to the order of the coefficient

$$\begin{array}{c}\hfill {\alpha }_k\left(t\right)=\sum _{j=0}^k{\beta }_j^{\left(k\right)}{t}^j.\end{array}$$

(2)

The general solution of ${\beta }_j^{\left(k\right)}$ can be calculated by a least squares adjustment using the system of equations

$$\begin{array}{c}\hfill \left. [\begin{array}{c}{𝒮}_p\\ {𝒮}_{p+1}\\ {𝒮}_{p+2}\\ ...\\ {𝒮}_{p+n}\end{array}\right]=\mathit{T}\mathit{M}\beta ,\mathrm{for}n\ge \frac{p^2+3p}{2},\mathrm{with}\end{array}$$

$$\begin{array}{cc}\hfill \mathit{T}& =\left. [\begin{array}{ccccc}{𝒮}_{p-1}& {𝒮}_{p-2}& {𝒮}_{p-3}& ...& {𝒮}_0\\ {𝒮}_p& {𝒮}_{p-1}& {𝒮}_{p-2}& ...& {𝒮}_1\\ ...\\ {𝒮}_{n-1}& {𝒮}_{n-2}& {𝒮}_{n-3}& ...& {𝒮}_{n-p}\end{array}\right],\hfill \\ \hfill \mathit{M}& =\left. [\begin{array}{ccccc}1& 0& 0& ...& 0\\ 0& 1& 0& ...& 0\\ 0& 0& 1& ...& 0\\ & & & \ddots & \\ 0& 0& 0& ...& 1\end{array}\left|\begin{array}{ccccc}t& 0& 0& ...& 0\\ 0& t& 0& ...& 0\\ 0& 0& t& ...& 0\\ & & & \ddots & \\ 0& 0& 0& ...& t\end{array}\right|\left.\begin{array}{cccc}0& 0& ...& 0\\ t^2& 0& ...& 0\\ 0& t^2& ...& 0\\ & & \ddots & \\ 0& 0& ...& t^2\end{array}\right|\begin{array}{ccc}0& ...& 0\\ 0& ...& 0\\ t^3& ...& 0\\ & \ddots & \\ 0& ...& t^3\end{array}\left|\begin{array}{c}\\ \\ \cdots \\ \\ \end{array}\right|\begin{array}{c}0\\ 0\\ 0\\ ⋮\\ t^p\end{array}\right]\mathrm{and}\hfill \\ \hfill \mathsf{\beta }& ={\left. [{\beta }_0^{\left(1\right)}{\beta }_0^{\left(2\right)}...{\beta }_0^{\left(p\right)}{\beta }_1^{\left(1\right)}{\beta }_1^{\left(2\right)}...{\beta }_1^{\left(p\right)}{\beta }_2^{\left(2\right)}...{\beta }_2^{\left(p\right)}...{\beta }_{p-1}^{(p-1)}{\beta }_{p-1}^{\left(p\right)}{\beta }_p^{\left(p\right)}\right]}^T.\hfill \end{array}$$

The order of the TVAR process is irrelevant for this calculation. Second, additional restrictions are put in place to ensure linear root movements. For orders 1 and 2, these restrictions were set out in the paper [11]. Now, this procedure is to be extended to the TVAR(3) process.

3. Restrictions of TVAR(3) Process with Linear Root Motion

The condition for the linear root movement arises from the transformation from the parameters ${\alpha }_k\left(t\right)$ (${\beta }_j^{\left(k\right)}$) to the roots $P_k\left(t\right)$, respectively. Therefore, the representation of the roots from the coefficients for the time-constant case is considered first.

3.1. Calculation of the Roots from the Time-Stable Coefficients

The roots of a third-order polynomial

$$\begin{array}{c}\hfill b\left(x\right)=x^3-{\alpha }_1\left(\tau \right)x^2-{\alpha }_2\left(\tau \right)x-{\alpha }_3\left(\tau \right)\end{array}$$

(3)

can be calculated after [13] [p. 22f] in a two-step procedure. The first step eliminates the monomial $x^2$ in (3) by substituting x with $y=x-\frac{{\alpha }_1\left(\tau \right)}{3}$. For the resulting polynomial over y

$$\begin{array}{c}\hfill b\left(y\right)=y^3+\underset{c_1\left(\tau \right)}{\underbrace{\left. [-\frac{{\alpha }_1^2\left(\tau \right)}{3}-{\alpha }_2\left(\tau \right)\right]}}y\underset{c_2\left(\tau \right)}{\underbrace{-2\frac{{\alpha }_1^3\left(\tau \right)}{27}-\frac{{\alpha }_1\left(\tau \right){\alpha }_2\left(\tau \right)}{3}-{\alpha }_3\left(\tau \right)}},\end{array}$$

(4)

the first roots ${\overline{P}}_1\left(\tau \right)$ of the Equation (4) can be determined using the Cardano solution:

$$\begin{array}{c}\hfill {\overline{P}}_1\left(\tau \right)=S_1+S_2,\end{array}$$

where

$$\begin{array}{cc}\hfill & S_1=\sqrt[3]{-\frac{c_2\left(\tau \right)}{2}+\sqrt{{\left(\frac{c_2\left(\tau \right)}{2}\right)}^2+{\left(\frac{c_1\left(\tau \right)}{3}\right)}^3}}\hfill \\ \hfill \mathrm{and}& S_2=\sqrt[3]{-\frac{c_2\left(\tau \right)}{2}-\sqrt{{\left(\frac{c_2\left(\tau \right)}{2}\right)}^2+{\left(\frac{c_1\left(\tau \right)}{3}\right)}^3}}.\hfill \end{array}$$

This root is back-substituted to find the roots of the characteristic polynomial of the third order in (3):

$$\begin{array}{c}\hfill P_1\left(\tau \right)={\overline{P}}_1\left(\tau \right)+\frac{{\alpha }_1\left(\tau \right)}{3}=S_1+S_2+\frac{{\alpha }_1\left(\tau \right)}{3}.\end{array}$$

The other two roots then result from

$$\begin{array}{cc}\hfill & P_2\left(\tau \right)=-\frac{S_1+S_2}{2}+\frac{{\alpha }_1\left(\tau \right)}{3}+i\sqrt{3}\frac{S_1-S_2}{2}\hfill \\ \hfill \mathrm{and}& P_3\left(\tau \right)=-\frac{S_1-S_2}{2}+\frac{{\alpha }_1\left(\tau \right)}{3}-i\sqrt{3}\frac{S_1-S_2}{2}.\hfill \end{array}$$

3.2. Restrictions for Linear Root Motion

These findings for a discrete point in time can be transferred to functions over time to derive the restrictions for linear root movements. However, this chapter does not derive the general condition for linear root motion; instead, the three sumands of $P_1\left(t\right)$

$$\begin{array}{ccc}\hfill S_1\left(t\right)=& \sqrt[3]{-\frac{c_2\left(t\right)}{2}+\sqrt{{\left(\frac{c_2\left(t\right)}{2}\right)}^2+{\left(\frac{c_1\left(t\right)}{3}\right)}^3}}\hfill & \hfill \stackrel{!}{=}{f}_1+g_{1}t,\end{array}$$

(5)

$$\begin{array}{ccc}\hfill S_2\left(t\right)=& \sqrt[3]{-\frac{c_2\left(t\right)}{2}-\sqrt{{\left(\frac{c_2\left(t\right)}{2}\right)}^2+{\left(\frac{c_1\left(t\right)}{3}\right)}^3}}\hfill & \hfill \stackrel{!}{=}{f}_2+g_{2}t\end{array}$$

(6)

$$\begin{array}{ccc}\hfill \mathrm{and}& -\frac{{\alpha }_1\left(t\right)}{3}\hfill & \hfill \stackrel{!}{=}{f}_3+g_{3}t\end{array}$$

(7)

are individually converted into linear functions by restrictions. Because of the three conditions, all linear combinations of these functions ($S_1$, $S_2$ and ${\alpha }_1$) are automatically linear. This particularly applies to $P_1\left(t\right)$, $P_2\left(t\right)$ and $P_3\left(t\right)$.

The conditions in (5), (6) and (7) have to be rewritten according to the conditions at the parameters ${\beta }_j^{\left(k\right)}$. After the derivation of the coefficients in (2), ${\alpha }_1\left(t\right)={\beta }_0^{\left(1\right)}+{\beta }_1^{\left(1\right)}t$ is already a polynomial of first degree and the condition (7) is always met. To simplify the conditions in (5) and (6), both conditions are potentiated by 3 and then linked to each other via addition or multiplication:

$$\begin{array}{cc}\hfill {(f_1+g_{1}t)}^3+{(f_2+g_{2}t)}^3& \stackrel{!}{=}-c_2\left(t\right)\mathrm{and}\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill (f_1+g_{1}t)(f_2+g_{2}t)& \stackrel{!}{=}-\frac{c_1\left(t\right)}{3}.\hfill \end{array}$$

(9)

This creates two new conditions: the first contains a third-order polynomial and the second a second-order polynomial, for which the equations are exactly satisfied if both sides have the same polynomial coefficients. Thus, the polynomial of order 3 in (8) includes four restrictions

$$\begin{array}{cc}\hfill f_1^3+f_2^3& =2{\left(\frac{{\beta }_0^{\left(1\right)}}{3}\right)}^3+\frac{{\beta }_0^{\left(1\right)}{\beta }_0^{\left(2\right)}}{3}+{\beta }_0^{\left(3\right)}\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill 3(f_1^2{g}_1+f_2^2{g}_2)& =2{\left(\frac{{\beta }_0^{\left(1\right)}}{3}\right)}^2{\beta }_1^{\left(1\right)}+\frac{{\beta }_0^{\left(1\right)}{\beta }_1^{\left(2\right)}+{\beta }_1^{\left(1\right)}{\beta }_0^{\left(2\right)}}{3}+{\beta }_1^{\left(3\right)}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill 3(f_1{g}_1^2+f_2{g}_2^2)& =2{\beta }_0^{\left(1\right)}{\left(\frac{{\beta }_1^{\left(1\right)}}{3}\right)}^2+\frac{{\beta }_0^{\left(1\right)}{\beta }_2^{\left(2\right)}+{\beta }_1^{\left(1\right)}{\beta }_1^{\left(2\right)}}{3}+{\beta }_2^{\left(3\right)}\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill g_1^3+g_2^3& =2{\left(\frac{{\beta }_1^{\left(1\right)}}{3}\right)}^3+\frac{{\beta }_1^{\left(1\right)}{\beta }_2^{\left(2\right)}}{3}+{\beta }_3^{\left(3\right)},\hfill \end{array}$$

(13)

and the polynomial of order 2 in (9) adds three further restrictions

$$\begin{array}{cc}\hfill f_1{f}_2& ={\left(\frac{{\beta }_0^{\left(1\right)}}{3}\right)}^2+\frac{{\beta }_0^{\left(2\right)}}{3}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill 3(f_1{g}_2+f_2{g}_1)& =2\frac{{\beta }_0^{\left(1\right)}{\beta }_1^{\left(1\right)}}{3}+{\beta }_1^{\left(2\right)}\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill g_1{g}_2& ={\left(\frac{{\beta }_1^{\left(1\right)}}{3}\right)}^2+\frac{{\beta }_2^{\left(2\right)}}{3}.\hfill \end{array}$$

(16)

With the help of the formulas (10), (13), (14) and (16), the variables

$$\begin{array}{cc}\hfill {\left(f_{1,2}\right)}^3& ={\left(\frac{{\beta }_0^{\left(1\right)}}{3}\right)}^3+\frac{{\beta }_0^{\left(1\right)}{\beta }_0^{\left(2\right)}}{6}+\frac{{\beta }_0^{\left(3\right)}}{3}\pm w_1\hfill \\ \hfill {\left(g_{1,2}\right)}^3& ={\left(\frac{{\beta }_1^{\left(1\right)}}{3}\right)}^3+\frac{{\beta }_1^{\left(1\right)}{\beta }_2^{\left(2\right)}}{6}+\frac{{\beta }_3^{\left(3\right)}}{3}\pm w_2\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill & w_1=\sqrt{{\left(\frac{{\beta }_0^{\left(3\right)}}{2}\right)}^2-{\left(\frac{{\beta }_0^{\left(2\right)}}{3}\right)}^3-\frac{{\left({\beta }_0^{\left(1\right)}{\beta }_0^{\left(2\right)}\right)}^2}{108}+\frac{{\left({\beta }_0^{\left(1\right)}\right)}^3{\beta }_0^{\left(3\right)}}{9}+\frac{{\beta }_0^{\left(1\right)}{\beta }_0^{\left(2\right)}{\beta }_0^{\left(3\right)}}{6}}\hfill \\ \hfill \mathrm{and}& w_2=\sqrt{{\left(\frac{{\beta }_3^{\left(3\right)}}{2}\right)}^2-{\left(\frac{{\beta }_2^{\left(2\right)}}{3}\right)}^3-\frac{{\left({\beta }_1^{\left(1\right)}{\beta }_2^{\left(2\right)}\right)}^2}{108}+\frac{{\left({\beta }_1^{\left(1\right)}\right)}^3{\beta }_3^{\left(3\right)}}{9}+\frac{{\beta }_1^{\left(1\right)}{\beta }_2^{\left(2\right)}{\beta }_3^{\left(3\right)}}{6}},\hfill \end{array}$$

can be determined. These results are used in the remaining restrictions (11), (12), and (15).

4. Application: Two GNSS Time Series

To test the theory, the time series of an altitude component of a GNSS station (shown in Figure 1) is used. The data were provided by the Federal Institute of Hydrology (BFG), which operates a GNSS monitoring network for georeferencing and monitoring selected measuring stations.

Before the TVAR estimation can be performed, data jumps, data holes and trend must be removed from the observations. To eliminate the jump in the time series, all observations on the left side of the data jump are reduced by their mean value and the same is carried out for the right-side observations. The data gaps are interpolated by third-order splines. In order to establish the stationarity of the time series, each observation is reduced by its predecessor:

$$\begin{array}{c}\hfill {\widehat{𝒮}}_t={𝒮}_t-{𝒮}_{t-1}.\end{array}$$

The offset of the resulting time series is eliminated by subtracting with the median of the time series. The result is shown in Figure 2.

In order to validate and compare the TVAR estimates later, we estimate time-stable AR(3) processes for a sliding window with the width of 100 observations. The roots of the AR(3) processes are shown in Figure 3. The time course goes from dark blue to light red, thus illustrating the temporal variability of the roots.

Now, the TVAR(3) process is estimated: On the one hand, this can be established via the direct method presented here for order 3 processes. The result is shown in Figure 4. Here, the linear movement of the roots is shown in the same color gradient as in Figure 3; the progress of the window had been displayed. For comparison, the roots from Figure 3 are shown as grey sequence in Figure 4 and Figure 5.

On the other hand, the successive estimation method from [11] was used to describe this time series. There are three ways to appreciate the TVAR(3) process:

• Via three TVAR(1) processes.
• Via a TVAR(1) process followed by a TVAR(2) process.
• Via a TVAR(2) process followed by a TVAR(1) estimate.

All three possibilities were realized and evaluated using the sum of squared residuals:

$$\begin{array}{c}\hfill SSR=\frac{{\mathit{e}}^T\mathit{e}}{n}.\end{array}$$

Here, $e_t={\widehat{𝒮}}_t-{\alpha }_1\left(t\right){\widehat{𝒮}}_{t-1}-{\alpha }_2\left(t\right){\widehat{𝒮}}_{t-2}-{\alpha }_3\left(t\right){\widehat{𝒮}}_{t-3}$ is the error that occurs in the TVAR(3) estimation, and n is the number of errors per estimate. Method 3 had the smallest SSR, and was therefore used for the comparison. The resulting root movement is again contrasted with the result from the windowing and is shown in Figure 5.

In Figure 4, it can immediately be noted that the root motion of the direct calculation moves longitudinally through the roots of the windowing, while the roots of the successive estimation in Figure 5 show little movement and do not lie in the middle of the roots of the windowed version. Over time, the beginning and end points of the successive estimation are quite accurate in the first and last window, but the course of time is almost completely overlooked. The time course of the direct estimation has been approximated well by its counterpart.

5. Conclusions and Outlook

In this elaboration, a method has been developed which allows for TVAR(3) processes with linear root movements to be turned into estimates without sequential procedures. For this purpose, the TVAR(3) estimation was extended by three nonlinear conditions to obtain linear root motions. In an application, it was shown that the solution obtained by the direct estimation of a TVAR(3) process with linear root movements better fits the data than the successive estimation of TVAR(1) and TVAR(2) processes. Therefore, the method set out here shows a useful extension of the TVAR process estimation with linear root motions. In further studies, the conditions for the TVAR(4) process can be determined to expand the evaluation possibilities. Polynomial degrees higher than 4 are not possible. This follows from the realization that there is no way to analytically determine the roots for these polynomials (see [14]).