ADMM-Based Differential Privacy Learning for Penalized Quantile Regression on Distributed Functional Data

Abstract

Alternating Direction Method of Multipliers (ADMM) is a widely used machine learning tool in distributed environments. In the paper, we propose an ADMM-based differential privacy learning algorithm (FDP-ADMM) on penalized quantile regression for distributed functional data. The FDP-ADMM algorithm can resist adversary attacks to avoid the possible privacy leakage in distributed networks, which is designed by functional principal analysis, an approximate augmented Lagrange function, ADMM algorithm, and privacy policy via Gaussian mechanism with time-varying variance. It is also a noise-resilient, convergent, and computationally effective distributed learning algorithm, even if for high privacy protection. The theoretical analysis on privacy and convergence guarantees is derived and offers a privacy–utility trade-off: a weaker privacy guarantee would result in better utility. The evaluations on simulation-distributed functional datasets have demonstrated the effectiveness of the FDP-ADMM algorithm even if under high privacy guarantee.

1. Introduction

Machine learning is becoming more and more common in statistical modeling and data analysis, along with the increasing concerns about the privacy disclosure of data. Therefore, we urgently need to develop algorithms which can provide privacy protection for personal data. In turn, the demand for data privacy protection has stimulated the establishment of formal standards on data privacy and the development of privacy framework. Among them, differential privacy (DP) [1,2] is the most widely discussed and developed technique in theory [3,4,5,6], and the feasibility of adopting these theories is shown among others by [7,8,9]. The framework of DP makes it convenient for us to construct privacy protection algorithms. However, these privacy protection algorithms may also need to pay a price of sacrificing the rate of convergence in statistical accuracy. Therefore, we need to develop differential privacy distributed learning algorithms that do not sacrifice statistical accuracy as much as possible for large-scale distributed data.

Distributed machine learning can disassemble the original huge training task into multiple sub-tasks, that is, transforming large-scale learning that one machine can not afford during collaborative learning with multiple machines. Recently, ref. [10] gave a sparse distributed learning solution for high-dimensional problems; ref. [11] proposed a distributed learning algorithm which segments features in a high-dimensional sparse additive model and proved the consistency of the sparse patterns for each additive component; ref. [12] provided a more flexible framework using the communication-efficient surrogate likelihood (CSL) procedure, which can solve different settings such as M-estimation for low- and high-dimensional problems and Bayesian inference. Ref. [13] extended the CSL method to distributed quantile regression and then established some statistical properties under quantile loss, which does not satisfy the smoothness condition of CSL method. In distributed learning, each sub-task is executed separately on an independent machine. It allows each machine to complete a collective learning objective, which is usually a standardized empirical risk minimization problem. The individual data that do not need to be disclosed will be calculated in a local iterative algorithm, and the parameters will be transferred between the central machine and each local machine. At present, the widely used algorithms for the decentralized distributed learning problems mainly include subgradient-based algorithms [14,15], alternating direction method of multipliers(ADMM) [16,17,18,19], and the combination algorithms of these methods [20]. Ref. [21] proved that ADMM-based algorithms converge at the rate of $O(1/L)$, while subgradient-based algorithms usually converge at the rate of $O(1/\sqrt{L})$, where L is the number of iterations. Therefore, in this paper, we adopt an ADMM-based distributed learning algorithm against privacy disclosure and keep a statistical guarantee.

We know that sensitive individual information may be leaked in optimization algorithms as a result of sharing information such as parameters and/or gradients of the model between machines, as presented in [22,23]. The same problem exists in our ADMM-based distributed algorithm: how to avoid privacy leakage. So, we need to protect privacy via a DP mechanism while maintaining statistical accuracy in our distributed learning. Ref. [24] studied a class of regularized empirical risk minimization machine learning problems via ADMM and proposed the dual variable perturbation and the primal variable perturbation methods for dynamic differential privacy. Ref. [25] proposed a privacy-preserving cooperative learning scheme, where users are allowed to train independently using their own data and only share some updated model parameters. They used an asynchronous ADMM approach to accelerate learning. In addition, their algorithm integrates secure computing and distributed noise generation to ensure the confidentiality of shared parameters during the asynchronous ADMM algorithm process. Ref. [26] applied a new privacy-preserving distributed machine learning (PS-ADMM) algorithm based on stochastic ADMM, which provides a privacy guarantee by perturbing the gradient and has a low computational cost. In the paper, we focus on a functional linear regression model for functional data analysis via ADMM-based distributing learning to keep DP and statistical efficiency.

Functional data are natural generalizations of multivariate data from finite dimensional to infinite dimensional which are obtained by observing a number of subjects over time, space, and other continua. In practice, functional data are frequently recorded by an instrument, which involves a large number of repeated measurements per subject. They can be curves, surfaces, images, or other complex objects; see some real data sets in the monographs [27,28,29]. All in all, a functional datum is not a single observation but rather a set of measurements along a continuum; taken together, they are regarded as a single entity, curve, or image. In recent decades, functional data analysis has drawn considerable attention because advanced technology makes functional data easier to collect in applied fields such as medical studies, speech recognition, biological growth, climatology, online auctions, and so on. Time series data are treated as multivariate data because they are given as a finite discrete time series. In addition, longitudinal data, which are often observed in biomedical follow-up studies, are strongly linked with functional data; however, their use often involves several (few) measurements per subject taken intermittently at different time points for different subjects. Therefore, functional data and longitudinal data are also intrinsically different. In addition, some classic multivariate data analysis tools, applied to time series and longitudinal data analysis, cannot be directly applied to functional data analysis because they ignore the fact that the underlying object of the measurements of a subject is a function such as curve or surface. We know that functional data are intrinsically infinite dimensional, and our analysis methods cannot be based on the assumption that the values observed at different times for a single subject are independent because of the intra-observation dependence. The high intrinsic dimensionality and the intra-observation dependence of functional data pose challenges both for theory and computation.

Recently, various approaches and statistical models for the analysis of functional data have been developed. For an introduction and summary, see [27,28,29,30]. Ref. [31] firstly proposed a linear regression model and analyzed the effects of functional independent variables on the scalar response variables through the inner product of functional independent variables and unknown nonparametric coefficient function. Ref. [32] gave a functional linear semiparametric quantile regression model, which has been used to analyze ADHD-200 patients data. Ref. [33] studied the estimation problem of a functional partial quantile regression model, and proved the asymptotic normality of the finite dimensional parameter estimation. The conventional method of functional data analysis is principal component analysis (PCA), such as [34,35]. Ref. [35] gave the optimal convergence rates of PCA. We will consider functional principal components analysis (FPCA) for our functional linear regression model and investigate the distributed learning with privacy.

In this paper, we propose a new ADMM-based distributed learning algorithm with differential privacy to handle large amounts of functional data. We call it the FDP-ADMM algorithm. Our proposed FDP-ADMM algorithm has good properties such as a faster rate of convergence, lower communication and computation costs, and better utility–privacy tradeoffs. In the FDP-ADMM algorithm, we consider a more robust quantile loss function, combine an approximate augmented Lagrange function, and integrate time-varying Gaussian noise into local learning on each machine. These techniques allow the FDP-ADMM algorithm to be adversarial while protecting privacy.

The main contributions of this paper are summarized as follows:

• We propose a distributed learning algorithm (FDP-ADMM) that can process large-scale distributed functional data and protect privacy. For the large-scale functional data, we adopt functional principal component analysis to reduce the dimensions of the data, improve the quality of data information, and promote the efficiency of functional data analysis using distributed learning.
• We introduce a quantile loss function for functional linear model such that our models are adaptive to heavy-tail data or outliers. Thus, our ADMM-based distributed learning algorithm is more robust compared with ordinary least square procedure.
• The privacy and theoretical convergence guarantees of the FDP-ADMM algorithm are derived, and a privacy–utility trade-off is demonstrated: a weaker privacy guarantee would result in better utility.
• We conduct numerical experiments to illustrate the effectiveness of FDP-ADMM in the framework of distributed learning. The results of experiments are consistent with our theoretical analysis.

The rest of this paper, is organized as follows. In Section 2, we state our problem formulation by introducing the functional linear regression model, the penalized quantile regression, the ADMM algorithm, and DP. In Section 3, we propose an ADMM-based distributed learning algorithm with privacy protection for distributed functional data analysis. In Section 4, we present the utility analysis of our algorithm, FDP-ADMM, including the convergence and privacy guarantee. In Section 5, we give some numerical experiments to verify our theoretical results. Some conclusions are given in Section 6. The proofs of the main results are collected in Appendix A.

Notations

For any positive integer n, we define $\left[n\right]:=\{1,2,\dots ,n\}$. $\parallel ·\parallel$, ${\parallel ·\parallel }_2$ and ${\parallel ·\parallel }_{\infty }$ are denoted as the Euclidean norm, ${\ell }_2$-norm, and ${\ell }_{\infty }$-norm, respectively. ${\rho }_{\tau }\left(u\right)=u(\tau -I\{u\le 0\})$ is the quantile loss function for a scalar $u\in \mathbb{R}$. and $I\{·\}$ denotes the indicator function. For a vector $\mathit{u}={(u_1,\cdots ,u_n)}^T$, we define ${\rho }_{\tau }\left(\mathit{u}\right)={\sum }_{i=1}^n{\rho }_{\tau }\left(u_i\right)$. Throughout this paper, the constant C denotes positive constant whose value may change from line to line. For any function f and a positive function $\varphi$, f$\asymp \varphi$ means $a\varphi <f<b\varphi$ for some positive constants a and b.

2. Problem Formulation

In this section, we present the functional data model, quantile regression, ADMM algorithm, and difference privacy mechanisms to be studied.

2.1. Functional Data Analysis

Functional data consist of functions that are basically smooth but are usually corrupted with noise, such as curves, images, and so on. For simplicity, we assume the functional predictor $X\left(t\right)$ on the finite time interval $I=[0,T].$ Let ${\left\{x_i\left(t\right):t\in I\right\}}_{1\le i\le n}$ be observation variables; that is, for each $t\in I$, there exists an observed value $x_i\left(t\right)\in \mathbb{R}$. A typical functional data set is:

$$\left\{x_i\left(t_{j,i}\right)\in \mathbb{R}:t_{j,i}\in I,1\le i\le n,1\le j\le J_i\right\},$$

where $J_i$ is the observation number, and n is the number of individuals. If $J_i$ is small, then the data are called sparse; otherwise they are called dense. FDA pays attention to the shape of the potential function or curve of the data via some statistical models and estimation procedures.

Functional linear regression is a standard method in functional data analysis for incorporating functional predictors, which focuses on modeling the relationship between a functional or continuous response Y and a functional predictor $X\left(t\right)$, in which t varies in a compact set I. It usually has the form:

$$Y={\beta }_0+{\int }_I\beta \left(t\right)X\left(t\right)dt+ϵ,$$

where ${\beta }_0$ is a intercept term, $ϵ$ is the random noise independent of $X\left(t\right)$, and $\beta \left(t\right)$ is an unknown function of interest. Without loss of generality, we assume $E\left(Y\right)=0$ and $E\left(X\right(t\left)\right)=0$. Based on data $\{(X_i\left(t\right),Y_i),i=1,\cdots ,n\}$, the model becomes:

$$Y_i={\int }_I\beta \left(t\right)X_i\left(t\right)dt+ϵ.$$

(1)

FPCA is commonly used for analyzing such models (1) with the purpose of dimension reduction. The main idea is to summarize the data variation and information via some dimensional loadings. Dimension reduction in FPCA is performed through an expansion of basis, which consists of the eigenfunctions formed by the covariance operator $\Sigma (·,·)$ of the process $X\left(t\right):t\in I$. By Mercer’s theorem, the spectral decomposition is

$$\Sigma (s,t)=Cov(X\left(s\right),X\left(t\right))=\sum _{k=1}^{\infty }{\lambda }_k{\varphi }_k\left(s\right){\varphi }_k\left(t\right),$$

where $s,t\in I$, ${\lambda }_k$ are the ordered eigenvalues such that ${\lambda }_1\ge {\lambda }_2\ge \dots$, and the function ${\varphi }_k$ forms the orthogonal basis corresponding to ${\lambda }_k$, $k=1,2,\cdots$.

By Karhunen and Loève [36], in the classical functional principal component analysis, the ith random curve $X\left(t\right)$ and functional coefficient $\beta \left(t\right)$ can be expressed as:

$$X_i\left(t\right)=\sum _{k=1}^{\infty }{A}_{ik}{\varphi }_k\left(t\right),\beta \left(t\right)=\sum _{k=1}^{\infty }{\mathit{w}}_k{\varphi }_k\left(t\right),$$

where the coefficients $A_{ik}={\int }_I{X}_i\left(t\right){\varphi }_k\left(t\right)dt$ and ${\mathit{w}}_k={\int }_I\beta \left(t\right){\varphi }_k\left(t\right)dt$ are the functional principal components. In addition, $E\left(A_{ik}\right)=0$ and var $\left(A_{ik}\right)={\lambda }_k$ for $k=1,2,\cdots$. We have the top K of ${\lambda }_k$ and taking their corresponding ${\varphi }_k\left(t\right)$, we have an approximation to $X_i\left(t\right)$ truncated as:

$$X_i^K\left(t\right)=\sum _{k=1}^K{A}_{ik}{\varphi }_k\left(t\right).$$

Then we consider the functional principal component regression into the functional linear model:

$$Y_i={\int }_I{X}_i\left(t\right)\beta \left(t\right)\mathrm{d}t+{ϵ}_i\approx \sum _{k=1}^K{A}_{ik}{\mathit{w}}_k+{ϵ}_{,}fori=1,\cdots ,n,$$

(2)

We regard ${\mathit{w}}_k$$(k=1,\cdots ,K)$ as unknown parameters. We can select a proper K FPCA basis to represent the functional data $X_i\left(t\right)$; that is, the most important information of data can be refined by FPCA.

2.2. Quantile Regression with Penalties

We have i.i.d observations $({\mathit{A}}_i,y_i),i=1,2,\cdots ,n$, with ${\mathit{A}}_i=(A_{i1},\cdots ,A_{iK})$. Let $Q_{y_i\mid {\mathit{A}}_i}\left(\tau \right)={\mathit{A}}_i{\mathit{w}}_{\tau }$ be the conditional quantile of $y_i$ on ${\mathit{A}}_i\in {\mathbb{R}}^K$, for a give the quantile level $\tau$th, $\tau$∈(0,1). Let ${\mathit{w}}_{\tau }={\left({\mathit{w}}_{1,\tau },\cdots ,{\mathit{w}}_{K,\tau }\right)}^T$. The quantile regression estimate of ${\mathit{w}}_{\tau }$ is defined as:

$${\widehat{\mathit{w}}}_{\tau }=\underset{\mathit{w}\in {\mathbb{R}}^K}{argmin}{\rho }_{\tau }(\mathit{y}-\mathit{A}\mathit{w}),$$

(3)

where $\mathit{A}={\left[{\mathit{A}}_1,\dots {\mathit{A}}_n\right]}^T\in {\mathbb{R}}^{n\times K}$ is a matrix, ${\rho }_{\tau }\left(u\right)=u(\tau -I\{u\le 0\})$ is the quantile loss function for a scalar $u\in \mathbb{R}$ and $I\{·\}$ denotes the indicator function.

Penalized quantile regression (PQR) is formulated as

$$\underset{\mathit{w}}{min}{\rho }_{\tau }(y-A\mathit{w})+P_{\lambda }\left(\mathit{w}\right),$$

(4)

where $P_{\lambda }(·)$ is a penalty, such as lasso penalty [37],

$$P_{\lambda }\left(\mathit{w}\right)=\lambda {∥\mathit{w}∥}_1;$$

(5)

or elastic net [38],

$$P_{\lambda }\left(\mathit{w}\right)=\lambda \left({\lambda }_2{∥\mathit{w}∥}_2^2+{\lambda }_1{∥\mathit{w}∥}_1\right),{\lambda }_1,{\lambda }_2\ge 0.$$

(6)

Penalized QR, such as (5) and (6), leads to biased estimators. To obtain an unbiased estimator, refs. [39,40] proposed a non-convex penalty, for instance, the MCP penalty [39], or the SCAD penalty [40].

Our learning empirical loss is:

$$\widehat{L}\left(\mathit{w}\right):=\frac{1}{n}\sum _{i=1}^n{\rho }_{\tau }\left(y_i-{\mathit{A}}_i\mathit{w}\right)$$

(7)

based on all data coming from all machines. As the data are distributed on local machines, it is difficult to collect all data into one machine, and additionally, there are privacy issues. Therefore, we apply the technique of distributed learning via the ADMM algorithm for the following distributed empirical loss (11). ADMM is a computational framework to solving optimization problems.

2.3. ADMM Algorithm

ADMM algorithm was first proposed by [41,42] in 1975 and 1976, respectively. Then, ADMM was reviewed and proven to be suitable for large-scale distributed optimization by [16]. In this section, we give the basic ADMM formulation.

Assume that our optimization problem is expressed as:

$$\underset{x,z}{min}\{f\left(\mathit{x}\right)+g\left(\mathit{z}\right)\}s.t.B\mathit{x}+C\mathit{z}=\mathit{d}$$

(8)

where $\mathit{x}\in {\mathbb{R}}^n,\mathit{z}\in {\mathbb{R}}^s$, matrices $B\in {\mathbb{R}}^{m\times n}$ and $C\in {\mathbb{R}}^{m\times s}$, vector $\mathit{d}\in {\mathbb{R}}^m$, and functions $f:{\mathbb{R}}^n\to \mathbb{R}$ and $g:{\mathbb{R}}^s\to \mathbb{R}$. x and z are the variables needing to be optimized. The optimization problem (8) consists of two parts: $f\left(\mathit{x}\right)$ related to variable x and $g\left(\mathit{z}\right)$ related to variable z. This structure can easily be dealt with via ADMM as follows: First, we have the augmented Lagrangian function:

$$L_{\rho }((\mathit{x},\mathit{z}),\mathit{u}):=f\left(\mathit{x}\right)+g\left(\mathit{z}\right)+{\mathit{u}}^T(B\mathit{x}+C\mathit{z}-\mathit{d})+\frac{\rho }{2}{\parallel B\mathit{x}+C\mathit{z}-\mathit{d}\parallel }_2^2,$$

(9)

where u is the dual variable (or called the Lagrange multiplier), and $\rho >0$ is a penalty parameter. The name ‘augmented’ in the $L_{\rho }$ refers to the quadratic penalty term $\frac{\rho }{2}{\parallel B\mathit{x}+C\mathit{z}-\mathit{d}\parallel }_2^2$, which is added for better convergence properties of algorithm. Then, the ADMM iterative solution of the optimization problem (9) is:

$$\begin{array}{cc}\hfill {\mathit{x}}^{l+1}& :=arg\underset{\mathit{x}}{min}{L}_{\rho }\left(\mathit{x},{\mathit{z}}^l,{\mathit{u}}^l\right),\hfill \\ \hfill {\mathit{z}}^{l+1}& :=arg\underset{\mathit{z}}{min}{L}_{\rho }\left({\mathit{x}}^{l+1},\mathit{z},{\mathit{u}}^l\right),\hfill \\ \hfill {\mathit{u}}^{l+1}& :={\mathit{u}}^l+\rho \left(B{\mathit{x}}^{l+1}+C{\mathit{z}}^{l+1}-\mathit{d}\right).\hfill \end{array}$$

(10)

The ‘multiplier method’ in ADMM refers to a dual ascent using augmented Lagrange function (with quadratic penalty term), and the ‘alternating direction’ refers to variables $\mathit{x}$ and $\mathit{z}$ be updated alternately. For more theories and applications about ADMM, refer to [16].

2.4. Differential Privacy

Differential privacy technology was originally designed to confront differential attacks problem. The traditional protecting method is to anonymize or encrypt to the datasets. However, some individual information can still be recovered from these anonymous data, based on certain algorithms, such as the recommendation algorithm. Therefore, Ref. [3] proposed the mechanism of differential privacy to protect privacy, which adds a designed noise in the algorithm so that attackers can not recover data information. Moreover, it has been proven that as long as the noise satisfies the differential privacy mechanism, no matter how much prior information the attacker has, the anonymous data cannot be reconstructed.

This paper mainly studies differential privacy for ADMM against adversarial attacks. Intuitively speaking, if an adversary can not tell whether a individual datum x belongs to the special data set X or not, when we output results from algorithm $\mathit{M}\left(\mathit{X}\right)$. We call that DP. Now, we give the definition of the $(ϵ,\delta )$-differential privacy from Dwork’s work [2].

Definition 1

($(ϵ,\delta )$-Differential Privacy). A randomized algorithm $\mathit{M}:{\mathit{X}}^n\to R$ is $(ϵ,\delta )$-differential private if for any two adjacent datasets (differing in only one tuple) X, $X^{\prime }\in {\mathit{X}}^n$, and for any measurable output subset $\mathit{S}\subseteq range\left(\mathit{M}\right)$:

$$\mathbb{P}[\mathit{M}\left(X\right)\in \mathit{S}]\le e^{ϵ}·\mathbb{P}\left[\mathit{M}\left(X^{\prime }\right)\in \mathit{S}\right]+\delta ,$$

where probability measure $\mathbb{P}$, which is bounded, only depends on the randomness of algorithm $\mathit{M}$.

In Definition 1, $\delta$ and $ϵ$ measure the protection strength of the privacy. It implies that a smaller $\delta$ or a smaller $ϵ$ gives better privacy protection. The Laplace and Gaussian mechanisms are two typical methods that are widely used in $(ϵ,\delta )$-differential privacy. They offer calibrated noise sampled from Laplace or Gaussian distribution, and add this noise into the algorithm. Now we consider a class of deferentially private algorithms via compositions, termed ‘k-fold adaptive composition’ in the literature. The advanced composition stated below, where the auxiliary inputs of the k-th algorithm are the outputs of all previous algorithms, shows how privacy parameters degrade as private algorithms are composited.

Lemma 1

(Theorem 4 in [43] (Advanced Composition)). Let $ϵ,\delta \ge 0$. The class of $(ϵ,\delta )$-differentially private algorithms satisfies $\left({ϵ}^{\prime },\delta \right)$-differential privacy under k-fold adaptive composition, where ${ϵ}^{\prime }=c_0\sqrt{k}ϵ$ for some constant $c_0$.

3. Distributed Learning with DP for Functional Data via ADMM

In this section, we propose the ADMM-based distributed learning algorithm with DP for functional data, which is called FDP-ADMM. We will transfer the functional regression model (1) into a linear regression model (2) by using FPCA. Because quantile regression has better performance in estimation and prediction for non-Gaussian distribution error, such as heavy-tailed distribution or outliers, we consider quantile regression for the models (1) and (2). First, based on data $\{(X_i\left(t\right),y_i),i=1,\cdots ,n\}$, the functional quantile linear regression model we consider is

$$y_i={\int }_I{\beta }_{\tau }\left(t\right)X_i\left(t\right)dt+{ϵ}_{\tau ,i},$$

where $\tau \in (0,1)$ is a given quantile level, and ${ϵ}_{\tau ,i}$ are random errors. Without loss of generality, we assume that the $\tau$th quantile of ${ϵ}_{\tau ,i}$ is equal to zero. The model can be written as $Q_{y_i|X_i}\left(\tau \right)={\int }_I{\beta }_{\tau }\left(t\right)X_i\left(t\right)dt$. Our goal is to learn the functional coefficient ${\beta }_{\tau }\left(t\right)$ by

$$\underset{{\beta }_{\tau }\left(t\right)}{min}\sum _{i=1}^n{\rho }_{\tau }\left(y_i-{\int }_I{\beta }_{\tau }\left(t\right)X_i\left(t\right)dt\right).$$

The problem is difficult because of the term ${\int }_I{\beta }_{\tau }\left(t\right)X_i\left(t\right)dt$. By the FPCA introduced in Section 2.1, we have the model (2):

$$y_i={\int }_I{X}_i\left(t\right){\beta }_{\tau }\left(t\right)\mathrm{d}t+{ϵ}_{\tau ,i}\approx \sum _{k=1}^K{A}_{ik}{w}_{\tau ,k}+{ϵ}_{\tau ,i},fori=1,\cdots ,n.$$

That is, functional quantile linear regression is transformed as an ordinary quantile linear regression. We suppress the dependency of $w_{\tau ,k}$ on $\tau$ for simplicity. Then, for the quantile linear regression, we propose penalized quantile regression learning, which is formulated as

$$\underset{\mathit{w}}{min}{\rho }_{\tau }(\mathit{y}-\mathit{A}\mathit{w})+P_{\lambda }\left(\mathit{w}\right).$$

It has been introduced in Section 2.2.

However, our dataset $\{(X_i\left(t\right),y_i),i=1,\cdots ,n\}$ cannot be collected on one machine, but distributed over M machines. That is, we have the distributed data $\{(X_{i,j}\left(t\right),y_{i,j}),i=1,\cdots ,M,j=1,\cdots ,m_i\}$, where M is the number of worker machines, and $m_i$ is the size of sample on the ith machine. Thus, based on FPCA, $\{(X_{i,j}\left(t\right),y_{i,j}),i=1,\cdots ,M,j=1,\cdots ,m_i\}$ is transformed as $\{(y_{ij},{\mathit{A}}_{i,j}),i=1,\cdots ,M,j=1,\cdots ,m_i\}$, where ${\mathit{A}}_{i,j}=(A_{i,j1},\cdots ,A_{i,jK})$ is the score of the jth sample on the ith worker machine. So, based on the distributed data $\{(y_{ij},{\mathit{A}}_{i,j}),i=1,\cdots ,M,j=1,\cdots ,m_i\}$ and the model (2), we have the following QR estimation in the distributed framework:

$$\begin{array}{c}\hfill {\widehat{\mathit{w}}}_{\tau }=argmin\sum _{i=1}^M\left(\sum _{j=1}^{m_i}\frac{1}{m_i}{\rho }_{\tau }\left(y_{ij}-{\mathit{A}}_{i,j}\mathit{w}\right)\right),\end{array}$$

(11)

where ${\rho }_{\tau }(·)$ is the loss function of quantile regression, and w is the unknown coefficient. Furthermore, we modify QR as penalized QR estimator for achieving faster shrinking, that is:

$$\begin{array}{c}\hfill {\widehat{\mathit{w}}}_{\tau }=argmin\sum _{i=1}^M\left(\sum _{j=1}^{m_i}\frac{1}{m_i}{\rho }_{\tau }\left(y_{ij}-{\mathit{A}}_{i,j}\mathit{w}\right)\right)+P_{\lambda }\left(\mathit{w}\right).\end{array}$$

(12)

Note that in (12), there exist two type of tuning parameters, K and $\lambda$, where K controls the number of scores to characterize the decomposition level of FPCA and $\lambda$ controls the fitness of the model.

When facing big data, that is, when n is very large, it is hard for one machine to learn w in (12). So, it is necessary to distributed storage and learning. Next, we will demonstrate the ADMM-based distributed learning for penalized QR. We provide a sketch of our FDP-ADMM algorithm based on the distributed data.

3.1. ADMM-Based Distributed Learning Algorithm

Assume we have M machines, and the ith machine has $m_i$ local data samples. Applying the ADMM algorithm, we re-formulate the problem (12) as:   

$$\begin{array}{cc}\hfill \underset{{\left\{{\mathit{w}}_i\right\}}_{i\in \left[M\right]}}{min}& \sum _{i=1}^M\left(\sum _{j=1}^{m_i}\frac{1}{m_i}{\rho }_{\tau }\left(y_{ij}-{\mathit{A}}_{i,j}{\mathit{w}}_i\right)+\frac{\lambda }{M}P\left({\mathit{w}}_i\right)\right),\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\mathit{w}}_i=\mathit{w},i=1,\dots ,M,\hfill \end{array}$$

(13)

where ${\mathit{w}}_i\in {\mathbb{R}}^K$ is the local model parameters, and $\mathit{w}\in {\mathbb{R}}^K$ is the global ones. Then, the augmented Lagrangian function for the ith machine is:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\rho ,i}\left({\mathit{w}}_i,\mathit{w},{\mathit{\gamma }}_i\right)=\sum _{j=1}^{m_i}& \frac{1}{m_i}{\rho }_{\tau }\left(y_{ij}-{\mathit{A}}_{i,j}{\mathit{w}}_i\right)+\frac{\lambda }{M}P\left({\mathit{w}}_i\right)-〈{\gamma }_i,{\mathit{w}}_i-\mathit{w}〉+\frac{\rho }{2}{∥{\mathit{w}}_i-\mathit{w}∥}^2\hfill \end{array}$$

(14)

$$\begin{array}{c}\hfill \mathrm{s}.\mathrm{t}.{\mathit{w}}_i=\mathit{w},i=1,2,\dots ,M.\end{array}$$

(15)

The objective (14) is decoupled and each worker only needs to minimize the sub-problem based on its local data set. Constraints (15) enforce all the local models to consensus. It results in the following iteration:

$$\begin{array}{c}\hfill {\mathit{w}}_i^l=\underset{{\mathit{w}}_i}{argmin}{\widehat{\mathcal{L}}}_{\rho ,i}\left({\mathit{w}}_i,{\mathit{w}}^{l-1},{\gamma }_i^{l-1}\right),\end{array}$$

(16)

$$\begin{array}{c}\hfill {\mathit{w}}^l=\frac{1}{M}\sum _{i=1}^M{\mathit{w}}_i^l-\frac{1}{M}\sum _{i=1}^M{\gamma }_i^{l-1}/\rho ,\end{array}$$

(17)

$$\begin{array}{cc}\hfill {\gamma }_i^l& ={\gamma }_i^{l-1}-\rho \left({\mathit{w}}_i^l-{\mathit{w}}^l\right).\hfill \end{array}$$

(18)

Note that each machine transfers its $\left({\mathit{w}}_i^l,{\mathit{\gamma }}_i^l\right)$ to a central machine. The central machine gathers them to update ${\mathit{w}}^l$ and then broadcasts it to each machine. Details for the algorithm are present in Algorithm 1. Based on output ${\mathit{w}}^L$, we obtain:

$${\widehat{\beta }}_{N-DP}\left(t\right)=\sum _{k=1}^K{\mathit{w}}_k^L{\varphi }_k\left(t\right).$$

Algorithm 1:ADMM for PQR of Functional Data (F-ADMM)

3.2. ADMM-Based Distributed Learning with DP

For achieving faster optimization, we make use of the first-order approximation to the penalized objective function. Then, ${\mathcal{L}}_{\rho ,i}\left({\mathit{w}}_i,\mathit{w},{\gamma }_i\right)$ in (14) becomes:

$$\begin{array}{ccc}\hfill {\widehat{\mathcal{L}}}_{\rho ,i}\left({\mathit{w}}_i,{\tilde{\mathit{w}}}_i^{l-1},\mathit{w},{\gamma }_i\right)& =& \sum _{j=1}^{m_i}\frac{1}{m_i}{\rho }_{\tau }\left(y_{ij}-{\mathit{A}}_{i,j}{\tilde{\mathit{w}}}_i^{l-1}\right)+\frac{\lambda }{M}{P}_{}\left({\tilde{\mathit{w}}}_i^{l-1}\right)\hfill \\ & & +〈\sum _{j=1}^{m_i}\frac{1}{m_i}{\mathit{\rho }}_{\tau }^{\prime }\left(y_{ij}-{\mathit{A}}_{i,j}{\tilde{\mathit{w}}}_i^{l-1}\right)+\frac{\lambda }{M}{P}_{}^{\prime }\left({\tilde{\mathit{w}}}_i^{l-1}\right),{\mathit{w}}_i-{\tilde{\mathit{w}}}_i^{l-1}〉\hfill \\ & & -〈{\gamma }_i,{\mathit{w}}_i-\mathit{w}〉+\frac{\rho }{2}{∥{\mathit{w}}_i-\mathit{w}∥}^2+\frac{{∥{\mathit{w}}_i-{\tilde{\mathit{w}}}_i^{l-1}∥}^2}{2{\eta }_i^l},\hfill \end{array}$$

(19)

where ${\eta }_i^l\in \mathbb{R}$ is the time-varying step size which decreases as the iteration l grows, ${\rho }_{\tau }^{\prime }$ is the subgradient of the quantile loss function, and $P_{}^{\prime }$ is the subgradient of the penalty. So, we have the following optimization problem:

$$\begin{array}{c}\hfill \underset{{\mathit{w}}_i}{min}\sum _{i=1}^M{\widehat{\mathcal{L}}}_{\rho ,i}\left({\mathit{w}}_i,{\tilde{\mathit{w}}}_i^{l-1},\mathit{w},{\gamma }_i\right)\mathrm{s}.\mathrm{t}.{\mathit{w}}_i=\mathit{w},i=1,2,\dots ,M.\end{array}$$

(20)

Here, we give the ADMM-based distributed learning algorithm with DF (FDP-ADMM) as follows:

$$\begin{array}{cc}\hfill {\mathit{w}}_i^l& =\underset{{\mathit{w}}_i}{argmin}{\widehat{\mathcal{L}}}_{\rho ,i}\left({\mathit{w}}_i,{\tilde{\mathit{w}}}_i^{l-1},{\mathit{w}}^{l-1},{\mathit{\gamma }}_i^{l-1}\right),\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill {\tilde{\mathit{w}}}_i^l& ={\mathit{w}}_i^l+{\xi }_i^l,\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill {\mathit{w}}^l& =\frac{1}{M}\sum _{i=1}^M{\tilde{\mathit{w}}}_i^l-\frac{1}{M}\sum _{i=1}^M{\mathit{\gamma }}_i^{l-1}/\mathit{\rho },\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill {\mathit{\gamma }}_i^l& ={\mathit{\gamma }}_i^{l-1}-\mathit{\rho }\left({\tilde{\mathit{w}}}_i^l-{\mathit{w}}^l\right),\hfill \end{array}$$

(24)

where ${\xi }_i^l$ in (22) are sampled from $\mathit{N}\left(0,{\sigma }_{i,l}^2{\mathbf{I}}_K\right)$, and ${\mathit{w}}^l$ in (23) is computed on the central machine. The rest are processed at each local machine. Details on FDP-ADMM algorithm are presented in Algorithm 2. Based on output ${\mathit{w}}^L$ of Algorithm 2, we obtain:

$${\widehat{\beta }}_{DP}\left(t\right)=\sum _{k=1}^K{\mathit{w}}_k^L{\varphi }_k\left(t\right).$$

Note that the central machine initializes the global ${\mathit{w}}^0$, while each worker machine initializes their own variables: the noisy primal variables $\left\{{\tilde{\mathit{w}}}_i^0\right\}$ and the dual variables $\left\{{\gamma }_i^0\right\}$ for $i\in \left[M\right]$. ${\xi }_i^l$ is Gaussian noise with zero-mean and variance ${\sigma }_{i,l}^2$, where ${\sigma }_{i,l}^2$ is obtained based on the Gaussian mechanism of DP, which is given in Theorem 1. Each worker machine updates its noisy primal variable ${\tilde{\mathit{w}}}_i^l$ based on (22). Then, the central machine receives all noisy primal variables ${\left\{{\tilde{\mathit{w}}}_i^l\right\}}_{i\in \left[M\right]}$ and the dual variables ${\left\{{\gamma }_i^l\right\}}_{i\in \left[M\right]}$ from the worker machine, and updates a global variable ${\mathit{w}}^l$. In addition, ${\mathit{w}}^l$ on central machine broadcasts to every worker machine to update the final dual variables ${\left\{{\gamma }_i^l\right\}}_{i\in \left[M\right]}$ using (24). It is an iterative cycle.

We set the variance ${\sigma }_{i,l}=\frac{2c_1\sqrt{2ln(1.25/\delta )}}{m_iϵ\left(\rho +1/{\eta }_i^l\right)}$ for obtaining the $(\delta ,ϵ)$-DP of the FDP-ADMM algorithm, which is set based on the Gaussian mechanism of DP. ${\sigma }_{i,l}^2$ is time-varying; that is, it decreases as iteration l increases. The motivation of using time-varying variance in the Gaussian mechanism is to reduce the negative impact of noise and ensure the convergence of the algorithm. We find that the negative impact will be mitigated by the method of decreasing noise and can achieve a stable solution.

For the communication and computation costs of our algorithms 1 and 2, here are some remarks. We know that it is unrealistic to send the estimator of $\beta \left(t\right)$, $t\in [0,1]$ on a worker machine to the central machine because it is infinite dimensional. In Algorithms 1 and 2, we only transmit the K-dimensional w in each round of communication, so the communication complexity is only $O\left(K\right)$. In practice of functional data analysis, K is usually a small number such as 5, 10, etc. Therefore, our algorithms are communication-efficient because of the low communication costs. In addition, in each round of learning, each worker machine learns its own low-dimensional parameter w based on its local data, then the central machine is responsible for summarizing these parameters from worker machines. This working mechanism greatly reduces the computational costs.    

Algorithm 2:ADMM-based distributed learning with DP for PQR of Functional Data (FDP-ADMM)

4. Utility Analysis

4.1. Privacy Guarantee

In the section, we will analyze the privacy guarantee of the proposed FDP-ADMM algorithm. During traditional parameter transmission, the shared information ${\left\{{\mathit{w}}_i^l\right\}}_{l\in \left[L\right]}$ can divulge the sensitive messages of original data. So, it is necessary to show outputs ${\left\{{\tilde{\mathit{w}}}_i^l\right\}}_{l\in \left[L\right]}$ with differential privacy.

Denote the two neighboring datasets ${\mathcal{A}}_i$ and ${\mathcal{A}}_i^{\prime }$. So, the ${\mathit{w}}_{i,{\mathcal{A}}_i}^l$ and ${\mathit{w}}_{i,{\mathcal{A}}_i^{\prime }}^l$ are the primal variables obtained from every local worker machine. From the FDP-ADMM algorithm, we add noise to ${\mathit{w}}_i^l$ by Gaussian mechanism. A fundamental tool used in DP is sensitivity. We use $l_2$-norm sensitivity. Due to the application of first-order approximation in the augmented Lagrange function, the proposed algorithm does not require the smoothness and strong convexity assumptions to the objective function for proving the sensitivity.

First, we give a lemma, which gives an $l_2$-norm sensitivity of ${\mathit{w}}_i^l$ under the sub-gradient ${\ell }^{\prime }$ of loss function ℓ, bounded.

Lemma 2.

Assume that ${∥{\ell }^{\prime }(·)∥}_2\le c_1$. The $l_2$-norm sensitivity of the local primal variable ${\mathit{w}}_i^l$ update function is given by:

$$\underset{{\mathcal{A}}_1,{\mathcal{A}}_1^{\prime }}{max}∥{\mathit{w}}_{i,{\mathcal{A}}_i}^l-{\mathit{w}}_{i,{\mathcal{A}}_i^{\prime }}^l∥=\frac{2c_1}{m_i^2\left(\rho +1/{\eta }_i^l\right)}.$$

Its proof is given in Appendix A. Lemma 2 shows that the $l_2$ sensitivity of ${\mathit{w}}_i^l$ is affected by the time-varying ${\eta }_i^l$. We set ${\eta }_i^l$ as a decreasing function of l, so the $l_2$ sensitivity decreases with increasing l. That is, if $ϵ$ and $\delta$ is fixed, the added noise in the proposed algorithm will become smaller as the l increases. Therefore, the algorithm will be stably convergent in spite of adding the noise. Then, we show that Algorithm 2 guarantees $(ϵ,\delta )$-differential privacy.

Theorem 1.

Assume that $\parallel {\mathit{>A}}_{ij}{\parallel }_2\le c_1$, $i=1,\cdots ,M$ and $j=1,\cdots ,m_i$ in the model (11). Let $ϵ\in (0,1]$ be arbitrary and ${\xi }_i^k$ be the noise sampled from Gaussian mechanism with variance ${\sigma }_{i,k}^2$, where

$${\sigma }_{i,k}=\frac{2c_1\sqrt{2ln(1.25/\delta )}}{m_iϵ\left(\rho +1/{\eta }_i^k\right)}.$$

The FDP-ADMM guarantees $(ϵ,\delta )$-differential privacy. Specifically, for any neighboring datasets ${\mathit{A}}_i$ and ${\mathit{A}}_i^{\prime }$, for any output ${\tilde{\mathit{w}}}_i^k$, the following inequality always holds:

$$Pr\left[{\tilde{\mathit{w}}}_i^k\mid {\mathit{A}}_i\right]\le e^{ϵ}·Pr\left[{\tilde{\mathit{w}}}_i^k\mid {\mathit{A}}_i^{\prime }\right]+\delta .$$

Its proof is given in Appendix A.

4.2. Convergence of the FDP-ADMM Algorithm

The convergence of ADMM for convex problems has been widely studied in recent years. Under the requirement of high precision, the convergence of ADMM goes very slowly. However, under the requirement of medium precision, the convergence speed of ADMM is acceptable, and the global solution can be achieved by dozens of iterations. Furthermore, Ref. [44] showed that ADMM could attain a global linear convergence on strict convexity and Lipschitz gradient, especially when matrix B and C in (8) are full column rank. The ADMM framework is suitable for large-scale statistical learning problems. More convergence analysis of ADMM under convexity were studied by [45,46,47,48,49,50,51,52], and so on. Ref. [53] proposed an approximate ADMM algorithm to make it converge to the stable point with a large enough penalty parameter. Ref. [54] gave the convergence of quantile regression using ADMM for convex and non-convex penalties. We refer to [54] for the convergence of our FDP-ADMM algorithm.

We define ${\mathit{w}}^{*}$ as the optimal solution of (13) and $c_{\mathit{w}}={∥{\mathit{w}}^{*}∥}_2$. The convergence of the algorithm is based on the fact that the quantile loss function is convex and non-smooth. For simplicity of analysis, we define some notations as follows:

$$\begin{array}{cc}& f_i\left({\mathit{w}}_i\right)=\sum _{j=1}^{m_i}\frac{1}{m_i}{\rho }_{\tau }\left(y_{ij}-{\mathit{A}}_{i,j}{\mathit{w}}_i\right)+\frac{\lambda }{M}P\left({\mathit{w}}_i\right),\hfill \\ & {\overline{\mathit{w}}}^L=\frac{1}{L}\sum _{l=1}^L{\mathit{w}}^l,{\overline{\gamma }}_i^L=\frac{1}{L}\sum _{l=1}^L{\gamma }_i^l,{\overline{\mathit{w}}}_i^l=\frac{1}{L}\sum _{l=0}^{L-1}{\tilde{\mathit{w}}}_i^l,\hfill \\ & {\mathit{u}}_i^l=\left[\begin{array}{c}{\tilde{\mathit{w}}}_i^l\\ {\mathit{w}}^l\\ {\gamma }_i^l\end{array}\right],{\mathit{u}}_i=\left[\begin{array}{c}{\mathit{w}}_i\\ \mathit{w}\\ {\gamma }_i\end{array}\right],F\left({\mathit{u}}_i^l\right)=\left[\begin{array}{c}-{\gamma }_i^l\\ {\gamma }_i^l\\ {\tilde{\mathit{w}}}_i^l-{\mathit{w}}^l\end{array}\right].\hfill \end{array}$$

We analyze the convergence of our proposed algorithm in terms of both the objective value and the constraint violation as [55]:

$$\sum _{i=1}^M\left(f_i\left({\overline{\mathit{w}}}_i^L\right)-f_i\left({\mathit{w}}^{*}\right)+g∥{\overline{\mathit{w}}}_i^L-{\overline{\mathit{w}}}^L∥\right),$$

where ${\sum }_{i=1}^M\left(f_i\left({\overline{\mathit{w}}}_i^L\right)-f_i\left({\mathit{w}}^{*}\right)\right)$ is used to measure the distance between the current objective value and the optimal value, and ${\sum }_{i=1}^{M}g∥{\overline{\mathit{w}}}_i^L-{\overline{\mathit{w}}}^L∥$ depicts the difference between the local model and the global one. If the training result of our FDP-ADMM algorithm achieves optimal and local models, it obtains consensus.

Lemma 3

([55], lemma 2). Assume ${\rho }_{\tau }(·)$ and $P(·)$ are convex. For any $l\ge 1$, we have:

$$\begin{array}{cc}& \sum _{i=1}^M\left(f_i\left({\tilde{\mathit{w}}}_i^{l-1}\right)-f_i\left({\mathit{w}}_i\right)+{\left({\mathit{u}}_i^l-{\mathit{u}}_i\right)}^{\top }F\left({\mathit{u}}_i^l\right)\right)\hfill \\ \hfill \le \sum _{i=1}^M& \left(\frac{{\eta }_i^l}{2}{∥f_i^{\prime }\left({\tilde{\mathit{w}}}_i^{l-1}\right)-\left(\rho +1/{\eta }_i^l\right){\xi }_i^l∥}^2-\frac{\rho }{2}{∥{\mathit{w}}_i-{\mathit{w}}^l∥}^2+\frac{\rho }{2}{∥{\mathit{w}}_i-{\mathit{w}}^{l-1}∥}^2\right.\hfill \\ & -\left(\rho +1/{\eta }_i^l\right)〈{\xi }_i^l,{\mathit{w}}_i-{\tilde{\mathit{w}}}_i^{l-1}〉+\frac{1}{2{\eta }_i^l}{∥{\mathit{w}}_i-{\tilde{\mathit{w}}}_i^{l-1}∥}^2-\frac{1}{2{\eta }_i^l}{∥{\mathit{w}}_i-{\tilde{\mathit{w}}}_i^l∥}^2\hfill \\ & \left.+\frac{1}{2\rho }{∥{\gamma }_i-{\gamma }_i^{l-1}∥}^2-\frac{1}{2\rho }{∥{\gamma }_i-{\gamma }_i^l∥}^2\right).\hfill \end{array}$$

Based on Lemma 3, we have:

Theorem 2.

Assume that $\parallel {\mathit{A}}_{ij}{\parallel }_2\le c_1$, $i=1,\cdots ,M$, and $j=1,\cdots ,m_i$ in the model (11); $P(·)$ are convex; and $∥P^{\prime }(·)∥\le c_2$. The domain of the dual variable is bounded, namely, $∥{\gamma }_i∥\le g$. We set the learning rate as:

$${\eta }_i^l=\frac{c_{\mathit{w}}}{\sqrt{2l}}{\left({\left(c_1+\lambda c_2/M\right)}^2+\frac{8K{c}_1^{2}ln(1.25/\delta )}{m_i^2{ϵ}^2}\right)}^{-\frac{1}{2}}.$$

For any $L\ge 1$ and g, we have:

$$\begin{array}{cc}& \mathbb{E}\left[\sum _{i=1}^M\left(f_i\left({\overline{\mathit{w}}}_i^L\right)-f_i\left({\mathit{w}}_i^{*}\right)\right.\right.+g∥{\overline{\mathit{w}}}_i^L-{\overline{\mathit{w}}}^L∥\hfill \\ \hfill \le & \sum _{i=1}^M\frac{c_{\mathit{w}}}{\sqrt{L}}\sqrt{2{\left(c_1+\lambda c_2/M\right)}^2+\frac{16K{c}_1^{2}ln(1.25/\delta )}{m_i^2{ϵ}^2}}+\frac{M\left(\rho c_{\mathit{w}}^2+g^2/\rho \right)}{2L}.\hfill \end{array}$$

Its proof is given in Appendix A. Theorem 2 shows our approach achieves the rate of convergence at $O(1/\sqrt{L})$, and gives an explicit utility-privacy trade-off of our FDP-ADMM algorithm. For the larger $ϵ$ or $\delta$, our algorithm has better utility. Note that the larger $ϵ$ or $\delta$ means the weaker privacy-preserving ability.

5. Simulation Study

In this section, we illustrate the performance of the proposed privacy-protection FDP-ADMM algorithm using a simulated study.

The simulation design is described as follows:

$$y_i={\int }_I\beta \left(t\right)X_i\left(t\right)dt+{ϵ}_{\tau i},i=1,\cdots ,n,$$

where n is the sample size on all worker machines;

$$\beta \left(t\right)=\sum _{k=1}^{50}{\mathit{w}}_k{\varphi }_k\left(t\right),$$

with $w_1=0.3$, $w_k=4{(-1)}^{k+1}{k}^{-2}$ for $k\ge 2$, ${\varphi }_1\left(t\right)\equiv 1$ and ${\varphi }_k\left(t\right)=2^{1/2}cos\left((k-1)\pi t\right)$ for $k\ge 2$;

$$X_i\left(t\right)=\sum _{k=1}^{50}{A}_{ik}{\varphi }_k\left(t\right),$$

where $A_{ik}$’s are independent and normal $N(0,k^{-2})$; and the errors

$${ϵ}_{\tau i}={ϵ}_i-F_{ϵ}^{-1}\left(\tau \right),$$

where $F_{ϵ}$ is the distribution function of ${ϵ}_i$, take $ϵ\sim t\left(3\right)$. Note that $F_{ϵ}^{-1}$ is subtracted from ${ϵ}_i$ to make the $\tau$th quantile of ${ϵ}_{\tau i}$ zero for identifiability. The datasets ${(y_i,X_i\left(t\right))}_{i=1}^n$ are distributed on M worker machines, and each machine has the same sample size m. So, $n=Mm$.

In the simulation, we set $n=\mathrm{100,000}$ samples, and randomly split the dataset into $M=10$, 20, and 50 groups to simulate the distributed learning condition. We take the penalty parameter $\rho =0.1$, and the regularized parameter $\lambda =0.05$. We set the level of quantile $\tau =\{0.1,0.25,0.5,0.75,0.9\}$, and set privacy budget per iteration $ϵ=\{0.01,0.05,0.1,0.2\}$ and $\delta =\left\{{10}^{-3},{10}^{-4},{10}^{-5},{10}^{-6}\right\}$. For each scenario, we run the algorithm 100 times. We consider our FDP-ADMM algorithm with typical $l_1$-norm and $l_2$-norm penalties and then assess it in terms of convergence and accuracy.

First, we report the mean integrated squared error (MISE) of the estimator ${\widehat{\beta }}_{DP}\left(t\right)$ computed on a grid of 100 equally spaced points on $I=[0,1]$, that is:

$$\mathrm{MISE}=\mathbb{E}\left({\int }_0^1{({\widehat{\beta }}_{DP}\left(t\right)-\beta \left(t\right))}^T({\widehat{\beta }}_{DP}\left(t\right)-\beta \left(t\right))dt\right).$$

Second, based on ${\overline{\mathit{w}}}_i^l$ and ${\overline{\mathit{w}}}^l$, we evaluate the convergence properties of the FDP-ADMM algorithm with respect to the augmented objective value, which measures the loss as well as the constraint penalty and is defined as:

$$\sum _{i=1}^M\left(f_i\left({\overline{\mathit{w}}}_i^l\right)+\rho ∥{\overline{\mathit{w}}}_i^l-{\overline{\mathit{w}}}^l∥\right).$$

Third, we evaluate the accuracy by empirical loss:

$$\frac{1}{M}\sum _{i=1}^M\sum _{j=1}^{m_i}\frac{1}{m_i}{\rho }_{\tau }\left(y_{ij}-{\mathit{A}}_{ij}{\tilde{\mathit{w}}}_i^k\right).$$

5.1. L1-Regularized Quantile Regression

We obtain the FDP-ADMM steps for the $l_1$-norm quantile regression by:

$$\begin{array}{cc}\hfill {\mathit{w}}_i^l& =\left(\frac{1}{m_i}\sum _{j=1}^{m_i}{A}_{i,j}^T(\tau -I_{\{y_{i,j}-A_{i,j}{\tilde{\mathit{w}}}_i^{l-1}\le 0\}})-\frac{\lambda }{M}sgn\left({\tilde{\mathit{w}}}_i^{l-1}\right)\right.\hfill \\ & \left.+{\gamma }_i^{l-1}+\rho {\mathit{w}}^{l-1}+{\tilde{\mathit{w}}}_i^{l-1}/{\eta }_i^l\right){\left(\rho +1/{\eta }_i^l\right)}^{-1},\hfill \\ \hfill {\tilde{\mathit{w}}}_i^l& ={\mathit{w}}_i^l+\mathcal{N}\left(0,{\sigma }_{i,l}^2{\mathbf{I}}_K\right),\hfill \\ \hfill {\mathit{w}}^l& =\frac{1}{M}\sum _{i=1}^M{\tilde{\mathit{w}}}_i^l-\frac{1}{M}\sum _{i=1}^M{\gamma }_i^{l-1}/\rho ,\hfill \\ \hfill {\gamma }_i^l& ={\gamma }_i^{l-1}-\rho \left({\tilde{\mathit{w}}}_i^l-{\mathit{w}}^l\right),\hfill \end{array}$$

where $sgn(·)$ is the sign function. Since the objective function is convex but non-smooth, we use Theorem 2 to set ${\eta }_i^l$ and apply Theorem 1 to set ${\sigma }_{i,l}$.

First, we list MISEs for the number of local machines, $M=10$, 20, 50, in

Table 1. MISEs of FDP-ADMM algorithm for $l_1$-regularized quantile regression when $M=10$.

				M = 10
$ϵ$	$\delta$	$\tau =0.1$	$\tau =0.25$	$\tau =0.5$	$\tau =0.75$	$\tau =0.9$
0.1	${10}^{-6}$	1.10935	1.08272	1.08042	1.06772	1.15352
	${10}^{-5}$	1.01472	0.88899	0.63939	0.72965	0.81495
	${10}^{-4}$	1.00092	0.50874	0.61940	0.42669	1.20186
	${10}^{-3}$	0.76701	0.51994	0.43749	0.45714	0.83579
0.2	${10}^{-6}$	0.88293	0.49352	0.37903	0.48573	1.13445
	${10}^{-5}$	0.86012	0.49398	0.35214	0.43233	0.93748
	${10}^{-4}$	0.85513	0.47709	0.40259	0.45489	0.89320
	${10}^{-3}$	0.97630	0.45829	0.36365	0.42839	0.95078
0.3	${10}^{-6}$	0.75955	0.40747	0.37103	0.43872	0.99364
	${10}^{-5}$	0.80688	0.40563	0.38785	0.43834	1.16647
	${10}^{-4}$	0.82462	0.43617	0.37086	0.42679	1.08880
	${10}^{-3}$	0.88212	0.43003	0.36211	0.43141	0.88249
0.5	${10}^{-6}$	0.94682	0.46687	0.37158	0.42812	0.96098
	${10}^{-5}$	0.92709	0.44007	0.38863	0.42447	0.89955
	${10}^{-4}$	0.93776	0.43648	0.36595	0.44708	0.89621
	${10}^{-3}$	0.95611	0.42511	0.37481	0.45199	0.98540
0.8	${10}^{-6}$	0.87232	0.44024	0.38308	0.44378	0.96025
	${10}^{-5}$	0.90369	0.44478	0.37863	0.44728	0.99718
	${10}^{-4}$	0.91670	0.43855	0.37769	0.45473	0.92953
	${10}^{-3}$	0.93121	0.43090	0.37537	0.43751	0.94244
∞	1	0.92588	0.43536	0.38291	0.44820	0.98044

,

Table 2. MISEs of FDP-ADMM algorithm for $l_1$-regularized quantile regression when $M=20$.

				M = 20
$ϵ$	$\delta$	$\tau =0.1$	$\tau =0.25$	$\tau =0.5$	$\tau =0.75$	$\tau =0.9$
0.1	${10}^{-6}$	9.14206	7.57861	8.63208	5.99993	7.25853
	${10}^{-5}$	4.91313	3.39025	3.84567	4.72525	2.98801
	${10}^{-4}$	4.07548	1.99489	2.15743	1.73728	3.75287
	${10}^{-3}$	1.19830	0.92899	0.89924	1.38734	1.72083
0.2	${10}^{-6}$	1.27889	1.07583	0.91118	1.38400	1.39675
	${10}^{-5}$	1.12262	0.64832	0.68967	0.60605	0.97722
	${10}^{-4}$	0.79600	0.48621	0.40788	0.32599	0.59192
	${10}^{-3}$	0.58956	0.35490	0.28023	0.36716	0.79028
0.3	${10}^{-6}$	0.67998	0.48955	0.39535	0.42677	0.69698
	${10}^{-5}$	0.62656	0.34088	0.31510	0.36501	0.82977
	${10}^{-4}$	0.64047	0.35214	0.23091	0.42389	0.52364
	${10}^{-3}$	0.64256	0.33683	0.31104	0.37006	0.65798
0.5	${10}^{-6}$	0.48045	0.37946	0.28304	0.38171	0.60391
	${10}^{-5}$	0.58620	0.35572	0.32657	0.35470	0.53452
	${10}^{-4}$	0.60473	0.32019	0.29540	0.35935	0.61225
	${10}^{-3}$	0.55453	0.35411	0.31989	0.36962	0.61693
0.8	${10}^{-6}$	0.62139	0.34965	0.28970	0.36232	0.61739
	${10}^{-5}$	0.56993	0.33311	0.32059	0.35161	0.52785
	${10}^{-4}$	0.59024	0.36645	0.31904	0.34726	0.60428
	${10}^{-3}$	0.54107	0.35753	0.32211	0.35959	0.61891
∞	1	0.57159	0.36387	0.34485	0.36757	0.60352

and

Table 3. MISEs of FDP-ADMM algorithm for $l_1$-regularized quantile regression when $M=50$.

				M = 50
$ϵ$	$\delta$	$\tau =0.1$	$\tau =0.25$	$\tau =0.5$	$\tau =0.75$	$\tau =0.9$
0.1	${10}^{-6}$	32.69092	45.61741	18.57951	42.89526	26.78259
	${10}^{-5}$	19.54404	18.28042	24.98649	14.56024	23.62843
	${10}^{-4}$	9.25744	15.35268	14.21395	16.51154	15.62429
	${10}^{-3}$	8.48043	8.02662	9.36117	8.54005	10.24176
0.2	${10}^{-6}$	7.30838	10.65070	6.65439	8.11806	8.28644
	${10}^{-5}$	3.28129	7.68274	5.46142	5.37228	6.88005
	${10}^{-4}$	3.69134	3.30237	2.53601	2.68905	4.13535
	${10}^{-3}$	1.23577	1.46382	0.89365	1.12239	1.55549
0.3	${10}^{-6}$	2.77899	2.96351	2.07031	2.92310	2.93987
	${10}^{-5}$	2.79987	2.22634	1.83420	1.72759	1.82370
	${10}^{-4}$	1.21077	0.99900	1.30057	0.93872	1.55615
	${10}^{-3}$	0.78847	0.53879	0.45738	0.51968	0.78215
0.5	${10}^{-6}$	0.95185	0.79408	0.94353	0.85746	1.39231
	${10}^{-5}$	0.91010	0.90503	0.54837	0.61216	0.83725
	${10}^{-4}$	0.69861	0.37440	0.44130	0.41332	0.55033
	${10}^{-3}$	0.47677	0.29099	0.26389	0.28654	0.50744
0.8	${10}^{-6}$	0.43862	0.42597	0.28514	0.41843	0.57740
	${10}^{-5}$	0.69838	0.32106	0.24453	0.33619	0.55950
	${10}^{-4}$	0.51937	0.39685	0.22664	0.28052	0.42013
	${10}^{-3}$	0.46601	0.26187	0.20617	0.24769	0.38190
∞	1	0.43715	0.30922	0.229919	0.29383	0.45229

, respectively. From Table 1, Table 2 and Table 3, we observe that (i) Our approach with larger $ϵ$ and larger $\delta$ has better convergence for all quantile levels because their MISEs are smaller, which also implies weaker privacy protection. When $ϵ=0.8$ and $\delta =0.001$, the MISEs of our FDP-ADMM algorithm with privacy policy are comparable to the ones of non-DP algorithm ($\delta =\infty$), that is, our FDP-ADMM algorithm does not sacrifice the estimation accuracy under weak privacy protection. (ii) For strong privacy protection, such as $ϵ=0.1$, the accuracy of our training model decreases as the number of machines increases. Because the size of the local dataset is smaller for a larger number of working machines, more noise should be added into the FDP-ADMM algorithm to obtain a higher privacy guarantee. For the large number of machines, a high estimation accuracy can be achieved by reducing privacy protection, for example, $M=50$. (iii) Our FDP-ADMM algorithm has a trade off between privacy and accuracy, i.e., the stronger the privacy protection, the lower the estimation accuracy. (iv) When $\tau =0.5$, this FDP-ADMM is a robust distributed learning algorithm for all parameters of privacy and number of machines we set. Because $\tau$ is farther from 0.5, its MISE is worse.

Second, we study the training performance (empirical loss) v.s. different number of distributed data sources under different levels of privacy protection when $\tau =0.5$. See Figure 1. Figure 1 shows that the accuracy of our training model will be reduced if more local machines are used. Since the number of agents is larger the smaller the size of the local dataset is, more noise should be added to guarantee the same level of differential privacy. Thus, it results in reducing the performance of the trained model. This is consistent with Theorem 1 that the standard deviation of noises is scaled by $1/mi$. From another perspective, when more local machines participate, a weaker privacy protection can obtain a higher estimation accuracy.

Third, we illustrate the convergence of the FDP-ADMM algorithm by demonstrating how the augmented objective value converges for different values of $ϵ$ and $\delta$. See Figure 2. Figure 2 shows our algorithm with larger $ϵ$ and $\delta$ (which implies the weaker privacy protection) has better convergence. This result is consistent with Theorem 2.

Finally, we evaluate the performance of FDP-ADMM by empirical loss for different levels of privacy protection. See Figure 3. Figure 3 shows our approach has fast convergence property for all privacy policies. In addition, all results we obtained show the privacy–utility trade-off of our FDP-ADMM: better utility is achieved when privacy leakage increases.

5.2. L2-Regularized Quantile Regression

We obtain the FDP-ADMM steps for $l_2$-norm quantile regression as follows:

$$\begin{array}{cc}\hfill {\mathit{w}}_i^l& =\left(\frac{1}{m_i}\sum _{j=1}^{m_i}{A}_{i,j}^T(\tau -I_{\{y_{i,j}-A_{i,j}{\tilde{\mathit{w}}}_i^{l-1}\le 0\}})-\frac{\lambda }{M}{\tilde{\mathit{w}}}_i^{l-1}+\right.\hfill \\ & \left.{\gamma }_i^{l-1}+\rho {\mathit{w}}^{l-1}+{\tilde{\mathit{w}}}_i^{l-1}/{\eta }_i^l\right){\left(\rho +1/{\eta }_i^l\right)}^{-1},\hfill \\ \hfill {\tilde{\mathit{w}}}_i^l& ={\mathit{w}}_i^l+\mathcal{N}\left(0,{\sigma }_{i,l}^2{\mathbf{I}}_K\right),\hfill \\ \hfill {\mathit{w}}^l& =\frac{1}{M}\sum _{i=1}^M{\tilde{\mathit{w}}}_i^l-\frac{1}{M}\sum _{i=1}^M{\gamma }_i^{l-1}/\rho ,\hfill \\ \hfill {\gamma }_i^l& ={\gamma }_i^{l-1}-\rho \left({\tilde{\mathit{w}}}_i^l-{\mathit{w}}^l\right).\hfill \end{array}$$

Similar to the setting of Section 5.1, we present results in

Table 4. MISEs of FDP-ADMM algorithm for $l_2$-regularized quantile regression when $M=10$.

				M = 10
$ϵ$	$\delta$	$\tau =0.1$	$\tau =0.25$	$\tau =0.5$	$\tau =0.75$	$\tau =0.9$
0.1	${10}^{-6}$	2.89210	2.75353	2.46941	2.54780	3.81696
	${10}^{-5}$	2.92584	2.27235	2.09434	2.27390	2.86815
	${10}^{-4}$	1.37434	1.52857	1.63519	1.01153	1.76823
	${10}^{-3}$	0.99495	0.95007	0.90349	1.12417	1.11879
0.2	${10}^{-6}$	1.14263	1.07720	0.69340	1.03047	1.25381
	${10}^{-5}$	0.89339	0.60944	0.76508	0.70292	1.22899
	${10}^{-4}$	0.76141	0.60331	0.43044	0.56315	0.87222
	${10}^{-3}$	0.61730	0.58038	0.38235	0.42983	0.67101
0.3	${10}^{-6}$	0.89291	0.52495	0.53238	0.50471	0.90524
	${10}^{-5}$	0.82594	0.52562	0.34734	0.48837	0.83485
	${10}^{-4}$	0.61622	0.43849	0.34253	0.47389	0.52728
	${10}^{-3}$	0.50265	0.34256	0.24852	0.36576	0.58658
0.5	${10}^{-6}$	0.50657	0.35774	0.33148	0.36034	0.61492
	${10}^{-5}$	0.62303	0.31707	0.25926	0.33131	0.56412
	${10}^{-4}$	0.44178	0.32944	0.27809	0.27520	0.56905
	${10}^{-3}$	0.55427	0.31399	0.22924	0.27581	0.53565
0.8	${10}^{-6}$	0.48112	0.30366	0.26190	0.29383	0.54639
	${10}^{-5}$	0.52303	0.31519	0.21492	0.30934	0.49795
	${10}^{-4}$	0.51200	0.29494	0.25546	0.26651	0.51737
	${10}^{-3}$	0.49710	0.32842	0.21990	0.26114	0.49119
∞	1	0.47430	0.27901	0.20853	0.26926	0.49421

,

Table 5. MISEs of FDP-ADMM algorithm for $l_2$-regularized quantile regression when $M=20$.

				M = 20
$ϵ$	$\delta$	$\tau =0.1$	$\tau =0.25$	$\tau =0.5$	$\tau =0.75$	$\tau =0.9$
0.1	${10}^{-6}$	12.09882	7.53753	8.99489	9.28500	10.41200
	${10}^{-5}$	6.36908	4.83324	9.44384	5.38515	7.07413
	${10}^{-4}$	5.40212	3.78114	2.92258	5.80128	3.78069
	${10}^{-3}$	3.56865	1.67093	2.02714	3.12041	2.73505
0.2	${10}^{-6}$	2.35128	2.40928	2.47491	3.13811	2.81266
	${10}^{-5}$	2.44151	1.86417	2.14190	1.79286	1.69856
	${10}^{-4}$	1.25333	1.12828	1.06031	1.29433	1.47163
	${10}^{-3}$	1.00380	0.67944	0.75994	0.89221	1.20207
0.3	${10}^{-6}$	1.44773	1.21530	1.82965	1.16274	1.67580
	${10}^{-5}$	0.92305	0.84215	0.98673	0.87402	0.83162
	${10}^{-4}$	0.61151	0.50056	0.62719	0.75269	1.11462
	${10}^{-3}$	0.79422	0.43239	0.46819	0.43958	0.65150
0.5	${10}^{-6}$	0.65224	0.49834	0.69298	0.50219	0.72637
	${10}^{-5}$	0.52623	0.48679	0.39195	0.41469	0.81617
	${10}^{-4}$	0.45604	0.44620	0.39356	0.35684	0.54628
	${10}^{-3}$	0.48589	0.35496	0.26164	0.28309	0.44583
0.8	${10}^{-6}$	0.58554	0.36776	0.36210	0.35751	0.44102
	${10}^{-5}$	0.56310	0.37512	0.24621	0.38069	0.41426
	${10}^{-4}$	0.46934	0.30216	0.22564	0.27554	0.43864
	${10}^{-3}$	0.39110	0.22099	0.22896	0.22580	0.42480
∞	1	0.37468	0.21974	0.16201	0.20135	0.38618

and

Table 6. MISEs of FDP-ADMM algorithm for $l_2$-regularized quantile regression when $M=50$.

				M = 50
$ϵ$	$\delta$	$\tau =0.1$	$\tau =0.25$	$\tau =0.5$	$\tau =0.75$	$\tau =0.9$
0.1	${10}^{-6}$	28.61208	40.72216	29.56713	19.82800	22.75443
	${10}^{-5}$	24.29355	10.61405	23.48655	25.28936	16.24641
	${10}^{-4}$	18.61518	10.56809	15.63120	18.87195	14.77067
	${10}^{-3}$	8.23236	11.80948	6.46954	8.75901	7.39285
0.2	${10}^{-6}$	9.21639	7.30661	10.83166	6.50947	6.74332
	${10}^{-5}$	5.33237	7.20520	6.48928	4.99331	6.98322
	${10}^{-4}$	3.69557	4.25720	5.23905	4.25916	4.26556
	${10}^{-3}$	2.64944	3.40341	2.10149	1.80372	2.19432
0.3	${10}^{-6}$	3.72135	5.26765	4.62668	2.82433	2.56538
	${10}^{-5}$	3.52995	3.49432	3.82902	2.46528	2.24935
	${10}^{-4}$	2.10102	2.30679	1.91362	2.48765	2.40823
	${10}^{-3}$	1.34345	1.65347	1.23016	1.02293	1.63715
0.5	${10}^{-6}$	1.76621	1.91785	1.38861	1.41912	2.15178
	${10}^{-5}$	1.14848	0.91837	1.25444	1.12365	0.96906
	${10}^{-4}$	0.82513	1.11108	0.79060	0.87009	1.03410
	${10}^{-3}$	0.63068	0.47302	0.41747	0.48054	0.59396
0.8	${10}^{-6}$	0.75022	0.86022	0.72234	0.83085	0.91919
	${10}^{-5}$	0.88104	0.91287	0.59528	0.52652	0.47342
	${10}^{-4}$	0.52061	0.44782	0.33773	0.43406	0.61304
	${10}^{-3}$	0.39031	0.37708	0.26519	0.28573	0.42317
∞	1	0.31062	0.18729	0.13762	0.16653	0.31931

and Figure 4, Figure 5 and Figure 6. We also obtain the same conclusion as in Section 5.1.

6. Conclusions

In the paper, we proposed an ADMM-based differential privacy learning algorithm on penalized quantile regression for functional data: FDP-ADMM. We first transform functional quantile regression into an ordinary linear regression model by functional principal analysis, and then design the FDP-ADMM algorithm by an approximate augmented Lagrange function, ADMM algorithm, and Gaussian mechanism with time-varying variance. The FDP-ADMM is a noise-resilient, convergent, and computationally effective distributed learning algorithm, even if for high privacy guarantee. Lastly, we obtain the estimation of coefficient function with privacy protection for functional quantile regression distributed model by the Karhunen and Loève expression. We also derived the privacy guarantee and theoretical convergence by the objective value and the constraint violation. The evaluations on simulation datasets have demonstrated the effectiveness of the FDP-ADMM algorithm, even if under high privacy protection, and have shown its privacy–utility trade-off: larger $ϵ$ and larger $\delta$, indicating weaker privacy guarantee, results in better utility.