Communication-Efficient Federated Learning with Adaptive Consensus ADMM

Abstract

This paper proposes utilizing federated learning (FL), a distributed learning paradigm, to process large, decentralized, and heterogeneous edge data in the context of Internet of Things (IoT) devices. However, heterogeneity and high communication costs are two primary challenges that hinder the efficacy of federated learning. To overcome these challenges, we have designed an algorithm, FedACADMM, which applies the adaptive consensus alternating direction method of multipliers (ACADMM) to federated learning clients (i.e., the edge mobile devices) to tackle the heterogeneity problem in federated networks. Importantly, the cost per round of communication for FedACADMM remains consistent with FedAvg and FedProx without adding any extra workload. Furthermore, our experimental results demonstrate that FedACADMM outperforms baseline methods with a realistic set of federated datasets, displaying enhanced convergence robustness. Notably, in highly heterogeneous scenarios, FedACADMM exhibits significant stability and rapid convergence, requiring at least 75% fewer communication rounds to achieve the target accuracy.

1. Introduction

The Internet of Things (IoT) has permeated many aspects of our daily lives through the proliferation of smart services and applications empowered by artificial intelligence (AI) [1]. HIS Markit estimates that by 2030, the number of IoT devices will reach 125 billion [2]. The massive number of IoT devices generates a huge amount of data that is aggregated by the industry in the cloud. Machine learning (ML) has evolved over the last few years as an effective methodology for handling complex data [3]. However, these IoT-generated data contain sensitive information about users or organizations, such as facial images, location information, health information, or fingerprint information. Uploading this information could result in information leakage. The issue of management and governance of data resources is the focus of many countries and government regulatory efforts. Countries and regions have introduced laws such as the General Data Protection Regulation (GDPR) [4] in the EU, the Personal Data Protection Act (PDP) [5] in Singapore, and the California Consumer Privacy Act (CCPA) [6] and the Consumer Privacy Bill of Rights (CPBR) [7] in the US to protect data privacy and security. This makes traditional centralized machine learning approaches, where distributed raw data generated on different devices or organizations are collected into a single server or a cluster with shared data storage, impossible.

Federated Learning (FL) is a novel distributed machine learning framework that allows user devices to benefit from models trained on rich data without the need to centrally collect these data. This approach safeguards the privacy and security of users. Federated learning comprises two parts: the edge mobile device (also known as the client) and the central server. In federated learning, the learning task is accomplished by a decentralized federation of clients that participate in federated learning, and these devices are coordinated by a central server. Each dataset of a client is locally trained and is not uploaded to the central server. Simultaneously, each client computationally updates the global model downloaded from the central server and sends the updated model back to the central server. The basic process of FL is described in [8] and shown in Figure 1.

 First, the model is trained locally, and then the parameters of the model are sent to the server. Finally, the cloud server returns the aggregated model to the local device. The above steps are repeated throughout the federated learning process until convergence. FedAvg is a method of simply averaging the locally uploaded models on the server [8]. Practically, FedAvg does not perform well when dealing with the complicated diversity of devices and data in the real world. Li et al. [9] proposed a federated optimization, FedProx, which is used to settle heterogeneity in FL. FedProx improves stability by adding a proximal term to the local training subproblem on every local device. Unfortunately, such a method did not meet the needs of various applications and did not obtain an improvement in convergence speed. FedMA is a new layer-wise FL algorithm for modern CNNs and LSTMs that appeal to Bayesian nonparametric methods to adapt to heterogeneity in the data [10]. However, this statistics-oriented aggregation method is susceptible to model poisoning attacks.

Primal-dual optimization method is a basic algorithm extensively employed in distributed learning. The alternating direction method of multipliers (ADMM) is a straightforward yet powerful approach that is particularly suitable for resolving distributed convex optimization issues, especially in the fields of applied statistics and machine learning [11]. ADMM takes on the shape of a decomposition-coordination technique, where solutions to small local subproblems are coordinated to discover solutions to large global problems. ADMM depends on a user-selected penalty parameter (step size) as its foundation. Theoretically, ADMM converges to any constant penalty parameter [12,13,14]; however, in practice, the efficiency of an ADMM is highly dependent on the selection of the parameter [15] and can be enhanced using adaptive penalty selection methods [16]. To summarize, taking into account the effectiveness of ADMM in distributed optimization, we have chosen to implement it in the context of federated learning (FL).

Several challenges differentiate optimization problems in federated learning, also known as federation optimization, from traditional distributed optimization. These challenges include user privacy, costly communication, and system and statistical heterogeneity [9]. The user privacy challenge arises because although federated learning protects data on devices by sharing model updates, communicating these updates throughout the training process can expose sensitive information to third parties and the central server [17]. The high cost of communication refers to the need for communication-efficient methods in the vast IoT network, where it is necessary to send small messages or model updates iteratively as part of the training process, instead of sending the entire dataset over the network, to adapt the model to the data generated by devices in the federated network [18]. System heterogeneity refers to differences in the storage, computational, and communication capabilities of IoT devices participating in federated learning. Statistical heterogeneity refers to differences in the way devices collect and generate data, as well as differences in the amount of data across devices. Clients participating in federated learning are typically highly heterogeneous in terms of both the size of their local datasets and computational speed [19]. Such challenges are common in real-world Internet of Things application scenarios. For example, there are many edge devices with computing power in the Internet of Things (these devices can be diverse, including wearable sensors, mobile phones, watches, and smart appliances). The data they generate are also diverse, and they are widely distributed across the Internet of Things; although they want to participate in federated learning training, they do not want their privacy to be compromised.

Expensive communication represents a significant bottleneck in federated learning, where a considerable number of clients are widely distributed, necessitating local data retention. In actual IoT communication networks, communication can be several orders of magnitude slower than local computation [20,21], thereby highlighting the urgent need for the development of communication-efficient methods. This approach considers two key aspects: (i) reducing the total number of communication rounds and (ii) decreasing the size of the transmitted information per round.

In this work, we propose a new federated optimization algorithm that combines adaptive consensus ADMM. This method successfully overcomes heterogeneity and converges in fewer communication rounds.

The main contributions of this paper are summarized as follows.

• We propose and demonstrate a communication-efficient federated optimization algorithm, which performs well in heterogeneous settings. The proposed method uses an adaptive consensus ADMM method to select local parameters. Instead of estimating a single global penalty parameter for all clients, different local penalty parameters are generated for each client based on the local gradient of the subproblems.
• We propose and adopt a new method similar to the double-rising and multiplier method to update the objective function of the customer. By using this method, the gradients of the server and the individual clients are aligned.
• By comparing with baselines in multiple popular datasets, we demonstrate that our federated optimization algorithm improves communication efficiency and is robust to client heterogeneity, especially in highly heterogeneous situations.

The rest of the paper is organized as follows. In Section 2, we provide an overview of related work. In Section 3, we review the core algorithmic idea of FL, and give the description of the proposed framework, FedACADMM. The convergence guarantee for FedACADMM is derived in Section 4 and the evaluation results are presented and analyzed in Section 5. Finally, Section 6 concludes the paper.

2. Related Work

Since FL was put forward [8], it has been committed to solving the security problems caused by the diversity and complexity of communication networks and the difficulties in obtaining private data [22]. In recent years, the research and discussion on federated optimization have been very active. Smith et al. [23] presented a federated multi-task learning approach for heterogeneity, MOCHA, which takes into account communication cost, stragglers, and error tolerance. This method is not suitable for non-convex models. The objective function of a non-convex model is not a convex function, which is a function where the straight line connecting any two points on the function lies completely above the function. Therefore, non-convex models can have multiple local minima and maximums, and their optimization problems are more complex. In non-convex settings, FedAvg is shown to be effective, but does not perform very well in the real-world situation of device heterogeneity [8]. FedProx modifies FedAvg by adding a weighted proximal term to the local objective function to account for heterogeneity [9]. It might be challenging to choose the weighted proximal term in FedProx for complicated purposes. In addition, subsequent research found that the answer to the question of whether FedProx can improve the speed of convergence is unclear [24]. A communication-efficient method, q-FFL, is inspired by techniques of fair resource allocation in wireless networks, and it alters the objective function of FedAvg by modifying the weights of different devices [25]. It performs well in both convex and non-convex models.

Numerous studies have been published that discuss the function of ADMM in distributed machine learning [11]. In [26], this idea derived from the R-ADMM as an application of the Peaceman–Rachford splitting. It demonstrated a virtually certain convergence of the proposed algorithm under broad assumptions about the distribution of communication loss and node activation events by utilizing findings from operator theory. The authors of [27] derived a perturbation strategy for ADMM in which the perturbed term is connected with the penalty parameters; it was shown that this simultaneously increases utility and privacy. The authors of [27] derived a perturbation strategy for ADMM, where the perturbed term is linked to the penalty parameters, which increases the utility and privacy at the same time.

The primal-dual methods are also widely used in FL. Based on the Peaceman–Rachford splitting approach, [28] developed FedSplit to solve the distributed convex problem and proved that FedSplit can solve FL problems under convexity only without any additional assumptions about the homogeneity of the system or data. FedSplit is only applicable to deterministic algorithms, excluding the random approximate gradient update method commonly utilized in practice. FedPD is a primitive dual-based meta-learning algorithm, which uses the dual variable to calculate the difference between local and server learning [29]. We note that all clients in FedPD are continuously participating in training. Ref. [30] considered both synchronous and asynchronous settings and applied the non-convex Douglas–Rachford splitting method, a random block coordinate strategy, and asynchronous updating to solve the non-convex composite optimization problem in federated learning. FedDR and asyncFedDR, unlike FedSplit and FedPD, only update a subset of customers at every round of updates and can update synchronously or asynchronously.

Our research was inspired by [31,32]. To increasing the robustness of federated optimization, in [31], FedADMM adds the local objective function with dual and augmented Lagrangian terms. It should be noted that this method works extremely effectively on large-scale networks with diverse data. Ref. [32] concentrated on a useful modification of the decomposition method by [16], allowing the penalty parameter to alter automatically based on specific adaptive principles. Unlike the original method, this method automatically varies the penalty parameters within an acceptable range and constructs appropriate adaptive rules. Combining the above methods, we propose the following solutions: (i) select only a subset of clients to participate in each iteration of the local update, (ii) adaptively update the penalty parameters before the local update, and (iii) formulate a suitable self-adaptive rule.

3. FedACADMM: Adaptive Consensus ADMM for Federated Learning

In this section, we introduce the key components of federated learning and ADMM, and then we formally present our proposed framework, FedACADMM. All the basic symbols throughout this article are listed in

Table 1. Symbols and Definitions.

Symbol	Definition
${\theta }^t$	The global model of the t-th round
$w_i$	The local model computed by client i
$S_t$	A subset of clients chosen at the t-th round
$\epsilon$	The stopping criterion
${\rho }^t$	The self-adaptive penalty parameter at the t-th round
$y_i$	The local dual variable kept by client i
${\delta }_i^t$	The local update at the t-th round by client i
$\eta$	The global step-size
t	The t-th round global iteration
${\tau }_i$	Hyperparameters for regulating the range of ${\rho }^{t+1}$ variations

.

3.1. Federated Optimization Problem

The classical federated learning problem involves constructing a single, global statistical model from numerous clients. The primary objective of federated learning is to enable the model to be learned using client-generated data while staying within the constraints of local storage and processing. During periodic updates, clients communicate only with the central server. The ultimate aim of federated learning is typically to minimize the following objective function:

$$\underset{w}{min}F\left(w\right),\mathrm{where}F\left(w\right):=\sum _{k=1}^m{p}_k{F}_k\left(w\right)$$

(1)

Here, m represents the total number of devices, $p_k\ge 0$, and ${\sum }_k{p}_k=1$. Additionally, $F_k$ represents the local objective function for the k-th device. The local objective function is commonly defined as the empirical risk computed over local data. The local objective function is used to optimize the parameters of the model by minimizing the empirical risk over the local data. The optimization is performed locally, and the results are then aggregated to obtain a global model that can be used for inference or further training. A typical local objective function used in federated learning is $F_k\left(w\right)=\frac{1}{n_k}{\sum }_{j_k=1}^{n_k}{f}_{j_k}\left(w;x_{j_k},y_{j_k}\right)$, where $n_k$ is the number of samples available locally. To adjust the weight of each device, a user-defined term $p_k$ is introduced, where $p_k$ is the weight assigned to device k and can be set to $\frac{1}{n}$ or $\frac{n_k}{n}$ depending on the desired effect. The total number of samples across all devices is denoted as $n={\sum }_k{n}_k$, which determines the scale of the learning task.

Convex objective refers to optimization problems in which the objective function is convex. Convex functions have the property that for any two points in their domain, the line segment connecting them lies on top of the graph of the function. Convex objectives are important in optimization because they have desirable mathematical properties that make them easier to optimize. For convex targets, the distributed local update primitive dual method utilizes a duality structure to effectively decompose the global target into sub-problems that can be solved in parallel in each round of communication. Hence, in this article, we decompose the objective function of federated learning into sub-problems using this method,

$$\underset{x\in {\mathbb{R}}^d}{min}\left\{F\left(x\right):=f\left(x\right)+g\left(x\right)=\frac{1}{n}\sum _{i=1}^m{f}_i\left(x\right)+g\left(x\right)\right\}$$

(2)

where m is the number of devices and the weighted local training loss is represented by $f_i$, which is considered to be L-smooth and nonconvex, and g is a proper, closed, and convex regularizer. We emphasize that the use of regularizers g has been emphasized in various works, including [33]. The goal of federated learning is to solve (2) for the model parameters $\theta$ using data distributed across the clients.

Let $dom\left(F\right):=\left\{x\in {\mathbb{R}}^d:F\left(x\right)<+\infty \right\}$ be the domain of F and $\partial g$ be the subdifferential of g [34]. Because (2) is nonconvex, we only anticipate discovering a stationary point, which meets the next optimality condition.

Definition 1. 

If $0\in \nabla f\left(x^{*}\right)+\partial g\left(x^{*}\right)$, then $x^{*}$ is called a first-order stationary point of (2).

Assumption 1. 

$dom\left(F\right)\ne \varnothing$ and $F^{*}:={inf}_{x\in {\mathbb{R}}^d}F\left(x\right)>-\infty$.

Assumption 2. 

Each local loss function $f_i(·)$ is L-Lipschitz smooth, i.e., $\forall x,y\in {\mathbb{R}}^d$, and the following inequalities exist:

$$∥\nabla f_i\left(x\right)-\nabla f_i\left(y\right)∥\le L\parallel x-y\parallel ,\forall x,y\in dom\left(f_i\right)$$

(3)

Federated optimization employs a local objective function based on individual device data as a proxy for the global objective function, thereby reducing communication.

3.2. Classical ADMM

ADMM is a powerful algorithm used for solving convex optimization problems in statistics and machine learning. This powerful technique decomposes the original problem into smaller subproblems that can be solved efficiently using dual ascent methods. After solving these subproblems, ADMM combines the obtained solutions using a multiplier method to achieve a high convergence speed. This algorithm solves the equality-constrained convex optimization problem,

$$\underset{x\in {\mathbb{R}}^n,z\in {\mathbb{R}}^m}{min}f\left(x\right)+g\left(z\right),s.t.Ax+Bz=c$$

(4)

where $f:{\mathbb{R}}^n\to \mathbb{R}$ and $g:{\mathbb{R}}^m\to \mathbb{R}$ are convex functions, $A\in {\mathbb{R}}^{p\times n}$, $B\in {\mathbb{R}}^{p\times m}$, and $c\in {\mathbb{R}}^p$. The Lagrangian for problem (4) is

$$\begin{array}{cc}\hfill \mathcal{L}(w,z,\pi )=f\left(w\right)+g\left(z\right)& +<Aw+Bz-c,\pi >\hfill \\ & +\frac{\sigma }{2}∥\begin{array}{c}Aw+Bz-c\end{array}∥,\hfill \end{array}$$

(5)

where $\sigma >0$ is called the penalty parameter. Augmented Lagrangian methods can bring robustness to the dual ascent method, especially in the absence of assumptions such as strict convexity or finiteness of f.

The iteration of ADMM can be written in the form

$$\left\{\begin{array}{cc}\hfill x^{k+1}& :=\underset{x}{argmin}{\mathcal{L}}_{\rho }(x,z^k,y^k)\hfill \\ \hfill z^{k+1}& :=\underset{z}{argmin}{\mathcal{L}}_{\rho }(x^{k+1},z,y^k)\hfill \\ \hfill y^{k+1}& :=y^k+\rho (A{x}^{k+1}+B{z}^{k+1}-c)\hfill \end{array}\right.$$

(6)

Here, $\rho >0$. By choosing an appropriate value for $\rho$, it can balance the rate of convergence and the feasibility of the iterates. The method, which consists of an x-minimization step, a z-minimization step, and a dual variable update, is very similar to dual ascent and the method of multipliers. The use of $\rho$ as a step size in dual updates ensures that the iterates $(x^{k+1},z^{k+1})$ remain dual feasible and leads to optimality as $A{x}^{k+1}+B{z}^{k+1}-c$ gradually converges to zero.

ADMM combines the decomposability of dual ascent with the superior convergence properties of multiplier methods, making it an efficient and widely used optimization tool.

3.3. FedACADMM Algorithm

We now turn to the description of a novel ADMM algorithm for federated optimization, and outline the FedACADMM method in Algorithm 1. Then, some discussions are offered to help throw light on the proposed method. The following straightforward yet crucial changes to FedACADMM lead to notable empirical advancements and also enable us to ensure convergence of the approach.

Algorithm 1 FedACADMM

Input: T, m, ${\rho }^t$, $S^t$, ${\theta }^t$, ${\theta }^{t-1}$, E, $\eta$ Output: ${\theta }^{t+1}$ 1: for$t=0,1,\dots ,T-1$do 2:    server selects a subset $S^t$ of m clients at random 3:    server broadcasts ${\theta }^t$ and ${\theta }^{t-1}$ to all chosen clients 4:    for client i $\in S^t$ do 5:      check the convergence verification and update the variable parameter ${\rho }^{t+1}$ as in (8) 6:      update client model as $w_i^{t+1}$ for E epochs as in (9) 7:      compute the difference ${\Delta }_i^t$ as in (10) 8:      sent ${\Delta }_i^t$ back to the server 9:     end for 10:   server calculates average parameter ${\theta }^{t+1}$ as in (11) 11: end for

We begin by describing the iterative process in detail. At the start of the t-th round, a subset of clients (denoted as $S_t$) is chosen, and each chosen client downloads the current server model ${\theta }^t$ and the last server model ${\theta }^{t-1}$. Confirm whether the convergence verification ${∥w_i-{\theta }^t∥}^2+{∥{\theta }^t-{\theta }^{t-1}∥}^2<\epsilon$ is satisfied, and stop the loop if it is satisfied. We give the definition of generalized residuals:

$$\left\{\begin{array}{cc}\hfill r^t& :=w_i^t-{\theta }^t\hfill \\ \hfill d^t& :={\theta }^t-{\theta }^{t-1}\hfill \end{array}\right.$$

(7)

Here, $r^t$ is the primal residual at iteration t, and $d^t$ is the dual residual at iteration t. We enable ${\rho }^{t+1}$ to fluctuate within an acceptable range from iteration to iteration:

$${\rho }^{t+1}=\left\{\begin{array}{cc}{\rho }^t/(1+{\tau }_i),\hfill & \mathrm{if}∥r^t∥<\mu ∥d^t∥\hfill \\ (1+{\tau }_i){\rho }^t,\hfill & \mathrm{if}\mu ∥r^t∥\ge ∥d^t∥\hfill \\ {\rho }^t,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(8)

with $\mu \in (0,1)$. This approach speeds up convergence by automating the local penalty parameter for each client, rather than relying on a global parameter for all clients. This approach eliminates the need to estimate a global penalty parameter, resulting in improved performance.

For the purpose of updating the client model locally:

$${\mathcal{L}}_i(w_i^t,y_i^t,{\theta }^t)=f_i\left(w_i^t\right)+{\left(y_i^t\right)}^{\top }(w_i^t-{\theta }^t)+\frac{{\rho }^{t+1}}{2}{∥w_i^t-{\theta }^t∥}^2$$

(9)

In the practical case, (9) does not need to be solved exactly, we can calculate the local model $w_i^t$ inexactly according to ${∥{\nabla }_{w_i}{\mathcal{L}}_i\left(w_i^t,y_i^t,{\theta }^t\right)∥}^2\le {\epsilon }_i$. Here, ${\epsilon }_i\ge 0$ represents the local accuracy. The use of the augmented Lagrangian ${\mathcal{L}}_i$ for training local model parameters $w_i^t$ has proven to be effective. This is due to the fact that a suitable value for the quadratic proximal coefficient $\rho$ can be determined, which allows the client accuracy level ${\epsilon }_i$ to be easily achieved.

The parameter difference uploaded to the server is:

$${\Delta }_i^t=(w_i^{t+1}+\frac{1}{{\rho }^{t+1}}{y}_i^{t+1})-(w_i^t+\frac{1}{{\rho }^{t+1}}{y}_i^t)$$

(10)

In our scenario, clients decide to perform different workloads to accommodate the heterogeneity of the system based on their local environment. In FedACADMM, each client holds a tuple of $w_i^t,y_i^t$, but it only transmits the enhanced model difference ${\Delta }_i^t$. This means that the communication cost per round is the same as in FedAvg and FedProx.

After collecting update messages from chosen clients, the server refreshes the global model as follows:

$${\theta }^{t+1}={\theta }^t+\frac{\eta }{\left|S^t\right|}\sum _{i\in S^t}{\Delta }_i^t$$

(11)

Here, $\eta >0$ is the global collecting step size. The inclusion of $\eta$ mitigates the effects of data heterogeneity and random algorithmic turbulence on the experimental outcomes, reducing their impact. The global model of the central server updates the current model using the selected subset of clients $S^t\in m$, which allows it to efficiently receive past information.

To date, a round of iterative updates has been accomplished.

4. Analysis

The goal of this section is to establish the global convergence for FedACADMM. We initially assert that the chosen stopping criterion is reasonable.

Let $\left\{(w_i,y_i^t,{\theta }^t)\right\}$ be the sequence created by our approach in the following analysis. We try to prove that $\left\{(w_i,y_i^t,{\theta }_t)\right\}$ converges to a solution point $\left\{(w_i^{*},y_i^{*},{\theta }^{*})\right\}$. We rewrite the task of FL to find the optimal parameter $\left\{(w_i^{*},y_i^{*},{\theta }^{*})\right\}$ that minimizes the overall loss,

$$\underset{w_i,{\theta }^t}{min}\sum _{i=1}^m{\alpha }_i{f}_i\left(w_i\right),\mathrm{s}.\mathrm{t}.w_i={\theta }^t,i\in \left[m\right].$$

(12)

we also denote $F\left(W\right):={\sum }_{i=1}^m{\alpha }_i{f}_i\left({\mathbf{w}}_i\right)$, where $F\left(w,w,...,w\right)=f\left(w\right)$. The solution point $\left\{(w_i^{*},y_i^{*},{\theta }^{*})\right\}$ is a stationary point of (12) if it satisfies,

$$\left\{\begin{array}{cc}\hfill a_i\nabla f_i\left(w_i^{*}\right)+y_i^{*}=0,& i\in \left[m\right]\hfill \\ \hfill w_i^{*}-{\theta }^{*}=0,& i\in \left[m\right]\hfill \\ \hfill \sum _{i=1}^m{y}_i^{*}=0\end{array}\right.$$

(13)

It is not difficult to prove that any locally optimal solution to problem (12) must satisfy (13). If $f_i$ is a convex function, then there is one and only one point that is a globally optimal solution when satisfying (13). Moreover, a solution point $\left\{(w_i^{*},y_i^{*},{\theta }^{*})\right\}$ of (12) indicates

$$\nabla f\left({\mathbf{w}}^{*}\right)=\sum _{i=1}^m{\alpha }_i\nabla f_i\left({\mathbf{w}}^{*}\right)=-\sum _{i=1}^m{\pi }_i^{*}=0$$

(14)

So, $w_i^{*}$ is also a solution point of

$${\mathbf{w}}_i^{*}:=\underset{\mathbf{w}\in {\mathbb{R}}^n}{argmin}f\left(\mathbf{w}\right)$$

(15)

Let us consider the differences between $\parallel y^t-y^{*}{\parallel }^2$ and $\parallel y^{t+1}-y^{*}{\parallel }^2$. Combine $\parallel y^t-y^{*}{\parallel }^2$ with the third equality of (7) and perform some modifications to get:

$$\begin{array}{cc}\hfill {∥y^t-y^{*}∥}^2=& {∥y^{t+1}-y^{*}∥}^2-{\gamma }^2{\left({\rho }^{t+1}\right)}^2{∥w_i^{t+1}-{\theta }^{t+1}∥}^2+2\gamma {\rho }^{t+1}{(y^t-y^{*})}^{\top }(w_i^{t+1}-{\theta }^{t+1})\hfill \end{array}$$

(16)

We provide some lemmas to help prove the main convergence theory and the following lemma provides a desirable property of the last term of (16).

Lemma 1. 

For $\forall \left\{(w_i^{*},y_i^{*},{\theta }^{*})\right\}$, we have

$$\begin{array}{cc}\hfill {(y^t-y^{*})}^{\top }(w_i^{t+1}-{\theta }^{t+1})& \ge {\rho }^{t+1}{∥w_i^{t+1}-{\theta }^{t+1}∥}^2+{\rho }^{t+1}\times {(w_i^{t+1}-w_i^{*})}^{\top }({\theta }^{t+1}-{\theta }^t)\hfill \end{array}$$

(17)

Proof of Lemma 1. 

For $\forall \left\{(w_i^{*},y_i^{*},{\theta }^{*})\right\}$, we have

$$\begin{array}{c}\hfill {(w_i^{t+1}-w_i^{*})}^T\left[f(w_i^{*})-y_i^{*}\right]\ge 0\end{array}$$

(18)

$$\begin{array}{c}\hfill {({\theta }^{t+1}-{\theta }^{*})}^T\left[g\left({\theta }^{*}\right)-y_i^{*}\right]\ge 0\end{array}$$

(19)

$$\begin{array}{c}\hfill w_i^{t+1}-{\theta }^{t+1}=0\end{array}$$

(20)

In the previous local update:

$${(w_i^{\prime }-w_i^{t+1})}^T\left\{f\left(w_i^{t+1}\right)-\left[y_i^t-{\rho }^{t+1}(w_i^{t+1}-{\theta }^t)\right]\right\}\ge 0$$

(21)

$$\begin{array}{c}\hfill {({\theta }^{\prime }-{\theta }^{t+1})}^T\left\{g\left({\theta }^{t+1}\right)-\left[y_i^t-{\rho }^{t+1}(w_i^{t+1}-{\theta }^{t+1})\right]\right\}\ge 0\end{array}$$

(22)

Similarly, we can get:

$$\begin{array}{c}\hfill {(w_i^{*}-w_i^{t+1})}^T\left\{f\left(w_i^{t+1}\right)-\left[y_i^t-{\rho }^{t+1}(w_i^{t+1}-{\theta }^t)\right]\right\}\ge 0\end{array}$$

(23)

$$\begin{array}{c}\hfill {({\theta }^{*}-{\theta }^{t+1})}^T\left\{g\left({\theta }^{t+1}\right)-\left[y_i^t-{\rho }^{t+1}(w_i^{t+1}-{\theta }^{t+1})\right]\right\}\ge 0\end{array}$$

(24)

We obtain by adding (18) and (23), and using the monotonicity of the operator f:

$${(w_i^{t+1}-w_i^{*})}^T\left[(y_i^t-y_i^{*})-{\rho }^{t+1}(w_i^{t+1}-{\theta }^t)\right]\ge 0$$

(25)

The same can be obtained by adding (19) and (24):

$${({\theta }^{t+1}-{\theta }^{*})}^T\left[(y_i^t-y_i^{*})-{\rho }^{k+1}(w_i^{t+1}-{\theta }^{t+1})\right]\ge 0$$

(26)

Now that (25) and (26) have been combined, together with (20), we have the statement of this lemma. □

So, we can obtain the inequality:

$$\begin{array}{cc}\hfill & {∥y^t-y^{*}∥}^2\ge {∥y^{t+1}-y^{*}∥}^2-\gamma (2-\gamma ){\left({\rho }^{t+1}\right)}^2{∥w_i^{t+1}-{\theta }^{t+1}∥}^2\hfill \\ \hfill & +2\gamma {\rho }^{t+1}{(w_i^{t+1}-w_i^{*})}^{\top }(w_i^{t+1}-{\theta }^{t+1})\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & {∥y^t-y^{*}∥}^2+\gamma {\left({\rho }^t\right)}^2{∥{\theta }^t-{\theta }^{*}∥}^2\ge {∥y^{t+1}-y^{*}∥}^2+\gamma {\left({\rho }^{t+1}\right)}^2{∥{\theta }^t-{\theta }^{*}∥}^2+\gamma (2-\gamma ){\left({\rho }^{t+1}\right)}^2\hfill \\ \hfill & \times {∥w_i^{t+1}-{\theta }^{t+1}∥}^2+\gamma {\left({\rho }^{t+1}\right)}^2{∥{\theta }^t-{\theta }^{t+1}∥}^2+2\gamma {\rho }^{t+1}{(w_i^{t+1}-{\theta }^{t+1})}^{\top }({\theta }^{t+1}-{\theta }^t)\hfill \end{array}$$

(28)

Lemma 2. 

For $t\ge 2$, we obtain

$$\begin{array}{cc}\hfill {\rho }^{t+1}& {(w_i^{t+1}-{\theta }^{t+1})}^{\top }({\theta }^{t+1}-{\theta }^t)\ge (1-\gamma ){\rho }^t{(w_i^t-{\theta }^t)}^{\top }({\theta }^{t+1}-{\theta }^t)\hfill \end{array}$$

(29)

We define $\gamma \in (0,(1+\sqrt{5})/2)$, and therefore, $1+\gamma -{\gamma }^2>0$. Assume $T=2-(1/3)(1+\gamma -{\gamma }^2)$, and we can obtain:

$$2-T=(1/3)(1+\gamma -{\gamma }^2)$$

(30)

$$T-\gamma =(1/3)({\gamma }^2-4\gamma +5)$$

(31)

Theorem 1. 

Let $\left\{w_i^{*},y^{*},{\theta }^{*}\right\}$ denote a solution to the variable inequality problem. Then, there exists a constant $t_0\ge 2$, if satisfied $\forall t\ge t_0$:

$$\begin{array}{cc}\hfill & {∥y^{t+1}-y^{*}∥}^2+\gamma {\left({\rho }^{t+1}\right)}^2{∥{\theta }^{t+1}-{\theta }^{*}∥}^2+\gamma (T-\gamma )\times {\left({\rho }^{t+1}\right)}^2{∥w_i^{t+1}-{\theta }^{t+1}∥}^2\le \hfill \\ \hfill & {(1+{\tau }_i)}^2({∥y^k-y^{*}∥}^2+\gamma {\left({\rho }^t\right)}^2{∥{\theta }^k-{\theta }^{*}∥}^2+\gamma (T-\gamma ){\left({\rho }^t\right)}^2{∥w_i^t-{\theta }^t∥}^2)-\gamma (2-T)\hfill \\ \hfill & \times {\left({\rho }^{t+1}\right)}^2({∥w_i^{t+1}-{\theta }^{t+1}∥}^2+{\left.∥{\theta }^{t+1}-{\theta }^t∥\right|}^2)\hfill \end{array}$$

(32)

Proof of Theorem 1. 

Combining (28), (29) and the Cauchy–Schwartz inequality, we obtain:

$$\begin{array}{cc}\hfill & {∥y^{t+1}-y^{*}∥}^2+\gamma {\left({\rho }_{t+1}\right)}^2{∥{\theta }^{t+1}-{\theta }^{*}∥}^2+\gamma (T-\gamma )\hfill \\ \hfill & \times {\left({\rho }^{t+1}\right)}^2{∥w_i^{t+1}-{\theta }^{t+1}∥}^2\le {∥y^k-y^{*}∥}^2\hfill \\ \hfill & +\gamma {\left({\rho }^t\right)}^2{∥{\theta }^k-{\theta }^{*}∥}^2+\gamma (T-\gamma ){\left({\rho }^t\right)}^2{∥w_i^t-{\theta }^t∥}^2\hfill \\ \hfill & -\gamma (2-T){\left({\rho }^{t+1}\right)}^2{∥w_i^{t+1}-{\theta }^{t+1}∥}^2-\gamma {\left({\rho }^{t+1}\right)}^2\hfill \\ \hfill & \times {\left.∥{\theta }^{t+1}-{\theta }^t∥\right|}^2+{\rho }^t{\rho }^{t+1}\left[\gamma {(1-\gamma )}^2\right]/(T-\gamma )\hfill \\ \hfill & \times {∥{\theta }^t-{\theta }^{t+1}∥}^2\hfill \end{array}$$

(33)

Assuming

$$[1/(1+{\tau }_i)]{\rho }^t\le {\rho }^{t+1}\le (1+{\tau }_i){\rho }^t$$

and ${lim}_{k\to \infty }{\tau }_k=0$ is established, after a series of transformations, we obtain Theorem 1. □

Theorem 2. 

Let the sequence $(w_i^t,y^t,{\theta }^t)$ produced by our method, we obtain:

$$\underset{t\to \infty }{lim}\left\{{∥w_i-{\theta }^t∥}^2+{∥{\theta }^t-{\theta }^{t-1}∥}^2\right\}=0$$

(34)

Theorem 3. 

Let the sequence $(w_i^t,y^t,{\theta }^t)$ produced by our method, we obtain:

$$\underset{t\to \infty }{lim}\left\{{∥w_i-{\theta }^t∥}^2+{∥{\theta }^t-{\theta }^{t-1}∥}^2\right\}=0$$

(35)

Proof of Theorem 3. 

From ${\tau }_i\ge 0$, ${\sum }_{i=0}^{\infty }{\tau }_i<\infty$ and ${\prod }_{n=1}^{\infty }(1+{\tau }_i)$, ${\sum }_{i=1}^{\infty }{\tau }_i^2$ are bounded. Suppose

$$C_s=\sum _{i=1}^{\infty }(2{\tau }_i+{\tau }_i^2),C_p=\prod _{i=1}^{\infty }(1+{\tau }_i)$$

It follows from Theorem 1 that, for $\forall t_0\in t$,

$$\begin{array}{cc}\hfill & {∥y^{t+1}-y^{*}∥}^2+\gamma {\left({\rho }^{t+1}\right)}^2{∥{\theta }^{t+1}-{\theta }^{*}∥}^2+\gamma (T-\gamma )\hfill \\ \hfill & \times {\left({\rho }^{t+1}\right)}^2{∥w_i^{t+1}-{\theta }^{t+1}∥}^2\le {(\prod _{t=t_0}^t(1+{\tau }_i))}^2({∥y^{t_0}-y^{*}∥}^2\hfill \\ \hfill & +\gamma {\left({\rho }^{t_0}\right)}^2{∥{\theta }^{t_0}-{\theta }^{*}∥}^2+\gamma (T-\gamma ){\left({\rho }^{t_0}\right)}^2{∥w_i^{t_0}-{\theta }^{t_0}∥}^2\left)\right)\hfill \\ \hfill & \le C_p^2({∥y^{t_0}-y^{*}∥}^2+\gamma {\left({\rho }^{t_0}\right)}^2{∥{\theta }^{t_0}-{\theta }^{*}∥}^2\hfill \\ \hfill & +\gamma (T-\gamma ){\left({\rho }^{t_0}\right)}^2{∥w_i^{t_0}-{\theta }^{t_0}∥}^2\left)\right)\hfill \end{array}$$

(36)

As a result, there exists a constant $C>0$ such that

$$\begin{array}{cc}\hfill & {∥y^{t+1}-y^{*}∥}^2+\gamma {\left({\rho }^{t+1}\right)}^2{∥{\theta }^{t+1}-{\theta }^{*}∥}^2+\gamma (T-\gamma )\hfill \\ \hfill & \times {\left({\rho }^{t+1}\right)}^2{∥w_i^{t+1}-{\theta }^{t+1}∥}^2\le C,\forall t\ge 0\hfill \end{array}$$

(37)

Combine (32) and (37), we obtain

$$\begin{array}{cc}\hfill & \sum _{t=t_0}^{\infty }\gamma (2-T){\left({\rho }^{t+1}\right)}^2({∥w_i^{t+1}-{\theta }^{t+1}∥}^2+{\left.∥{\theta }^{t+1}-{\theta }^t∥\right|}^2)\hfill \\ \hfill & \le {∥y^{t_0}-y^{*}∥}^2+\gamma {\left({\rho }^{t_0}\right)}^2{∥{\theta }^{t+1}-{\theta }^{*}∥}^2+\gamma (T-\gamma )\hfill \\ \hfill & \times {\left({\rho }^{t_0}\right)}^2{∥w_i^{t+1}-{\theta }^{t+1}∥}^2\sum _{t=t_0}^{\infty }(2{\tau }_i+{\tau }_i^2)({∥y^t-y^{*}∥}^2\hfill \\ \hfill & +\gamma {\left({\rho }^t\right)}^2{∥{\theta }^t-{\theta }^{*}∥}^2+\gamma (T-\gamma ){\left({\rho }^t\right)}^2{∥w_i^t-{\theta }^t∥}^2)\hfill \\ \hfill & \le (1+C_s)C\hfill \end{array}$$

(38)

and then

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}\gamma (2-T){\left({\rho }^{t+1}\right)}^2({∥w_i^{t+1}-{\theta }^{t+1}∥}^2+{∥{\theta }^{t+1}-{\theta }^t∥}^2)=0\end{array}$$

(39)

This means that the sequence $\left\{{\rho }^t\right\}$ is both an upper bound and a lower bound away from zero, which means that we have

$$inf{\left\{{\rho }^t\right\}}_1^{\infty }:=B_L>0,sup{\left\{{\rho }^t\right\}}_1^{\infty }:=B_U<\infty .$$

and from (30), Theorem 3 is proved. □

If the sequence $\left\{{\rho }^t\right\}$ is generated by (8), then it satisfies the following two conditions:

Condition C1.$inf{\left\{{\rho }^t\right\}}_1^{\infty }=B_L>0$ and ${\sum }_{t=1}^{\infty }{\eta }_t^2<+\infty$, where

$$\begin{array}{cc}\hfill & {\theta }_t=\left\{\begin{array}{cc}\sqrt{1-{\left({\rho }^t/{\rho }^{t+1}\right)}^2},\hfill & \mathrm{if}t\in T_D,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \\ \hfill & K_D:=\left\{t\mid {\rho }^{t+1}<{\rho }^t\right\}.\hfill \end{array}$$

(40)

Condition C2.${\sum }_{t=1}^{\infty }{\theta }_t^2<+\infty$, where

$$\begin{array}{cc}\hfill {\eta }_t& =\left\{\begin{array}{cc}\sqrt{{\left({\rho }^{t+1}/{\rho }^t\right)}^2-1},\hfill & \mathrm{if}t\in T_I,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \\ \hfill T_I& :=\left\{t\mid {\rho }^{t+1}>{\rho }^t\right\}.\hfill \end{array}$$

(41)

In this case, we can obtain

$$\begin{array}{cc}\hfill & t\in T_I=\left\{t\mid {\rho }^{t+1}>{\rho }^t\right\},\hfill \\ \hfill & t\in T_D=\left\{t\mid {\rho }^{t+1}<{\rho }^t\right\},\hfill \end{array}$$

(42)

So, it can be inferred that

$$\begin{array}{cc}\hfill & {\eta }_t^2=2{\tau }_t+{\tau }_t^2,\hfill \\ \hfill & {\theta }_t^2=2{\tau }_t+{\tau }_t^2.\hfill \end{array}$$

(43)

Thus,

$$\sum _{t=1}^{\infty }{\eta }_t^2<\infty \mathrm{and}\sum _{t=1}^{\infty }{\theta }_t^2<\infty .$$

(44)

Therefore, the proposed method using the parameter-adjusting strategy in (8) is convergent. Two sequences converge to the optimal function value of (15), namely,

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}F\left(W_t\right)=\underset{t\to \infty }{lim}f\left({\mathbf{w}}^t\right)=f^{*}\end{array}$$

(45)

Any accumulating point $\left(w_i^{\infty },y^{\infty },{\theta }^{\infty }\right)$ of sequence $\left\{w_i^{*},y^{*},{\theta }^{*}\right\}$ is an optimal solution to (12), where $w^{\infty }$ is an optimal solution to (15). Our algorithm implementation has been demonstrated to achieve both global and linear convergence, under the assumption that the Lagrange multipliers satisfy specific initialization conditions and provided that the local gradients are Lipschitz continuous.

5. Experiments

This section runs multiple numerical experiments to evaluate the performance of FedACADMM. We first describe our experimental setup including datasets, models, and data distribution in Section 5.1. We then demonstrate the effect of modifying the penalty parameters and compare our proposed experiments with several baseline methods in Section 5.2 and Section 5.3.

5.1. Experimental Setup

Datasets and Models. In our study, we adopted the standard federated configuration where every client owns their local data and communicates with the central server to transmit and receive information. Our experiments involved handwriting recognition and image classification using three well-known benchmark datasets: MNIST [35], Fashion MNIST (FMNIST) [36], and CIFAR-10 [37]. MNIST consists of handwritten digits from zero to nine. FMNIST includes images of 10 different categories of clothing, such as T-shirts, pants, and sneakers. CIFAR-10 is a dataset composed of 10 different types of objects, including aircraft, cars, dogs, and so on. To conduct these experiments, we utilized two CNN models, each comprising two convolutional layers, two fully connected layers, and a linear transformation layer. The detailed architecture of the models is provided in

Table 2. Datasets and models.

Datasets	Task	# Parameter	Model
MNIST	Handwritten recognition	1,663,370	2-layer CNN + FC
CIFAR-10	Image classification	1,105,098	2-layer CNN + FC
FMNIST	Image classification	1,663,370	2-layer CNN + FC

.

Data Distribution. To investigate federated optimization and conduct comparative experiments, we adopted the data distribution settings used in the FedAvg approach. Specifically, we conducted experiments using two different data division methods: IID (homogeneous) and non-IID (heterogeneous). Under the IID method, clients receive an equal distribution of data. Under the non-IID method, we arranged the training data by labels and then distributed them evenly among the shards. Each client was randomly and evenly assigned two shards.

Validation metrics. To assess the generalization performance of our approaches, we applied the entire test set in well-known federated datasets. Because convergence speed and final performance are important indicators of federated learning, we computed the performance attained at a specified number of rounds and the number of communication rounds required for the algorithm to reach the test target accuracy. Typically, the ADMM approach converges to a reasonable accuracy within a few tens of iterations. Therefore, we selected one hundred experimental rounds for all datasets after preliminary estimation. We acquired a reasonable evaluation target accuracy displayed in

Table 3. Target Accuracy. For better representation of figures and tables, we set different target accuracy for different data distributions in the same dataset.

	IID	Non-IID
MNIST	98%	96%
CIFAR-10	50%	45%
FMNIST	85%	80%

. For techniques that did not achieve the desired accuracy within the maximum communication round ($t>100$), we appended a plus sign (+) to the communication round. Additionally, we reported the communication round savings of FedACADMM compared to the classical FedAvg algorithm.

Implementation Details. We have implemented FedACADMM in PyTorch, which involves simulating a federated learning network with one server and m clients. To enable comparisons with other baselines, we reused the implementations of FedAvg and FedProx. All of our numerical experiments were conducted on a cloud computing platform, where we used a virtual machine with the following specifications: 4 vCPUs, 16 GB memory, and an NVIDIA RTX A4000 GPU. The operating system utilized for the training was Ubuntu 18.04 LTS, and we employed PyTorch 1.8.1 as our deep learning framework for training.

We set the global learning rate to 1 for all methods and used a client subset size equal to 10% of the total number of customers $(C=0.1)$. For all algorithms, we utilized the SGD optimizer with a learning rate of 0.1 for local updates.

5.2. Choosing the Initial Penalty Parameter

When designing the evaluation method for FedACADMM, the first consideration in this paper was determining the appropriate initial self-adaptive penalty parameter, denoted as ${\rho }^0$, which is crucial for (8). To achieve a balanced initial residual $∥w_i^t-{\theta }^t∥$ and pairwise residual $∥{\theta }^t-{\theta }^{t-1}∥$, an empirical rule-based heuristic optimization hyperparameter method was employed. This is because if one of the two terms approaches zero gradually, the algorithm would converge slowly. Another consideration in proposing the rule is to prevent additional computation. In particular, a practical approach in practice would be to run FedACADMM with various values of ${\rho }^0$ in parallel to obtain multiple final global models, and then select between these based on performance (e.g., accuracy) on the validation data. In this paper, we adopted an empirical rule-based heuristic optimization method to narrow down the search space and select a value for ${\rho }^0$ from a limited set of candidates, namely $\left\{{10}^{-3},{10}^{-2},0.1,1,10\right\}$, following the experimental setups of [9,32].

Figure 2 displays a comparison of test accuracy curves and training loss curves across various initial value penalty values, denoted as ${\rho }^0$. For clarity, we omit the curves associated with poor experimental results when ${\rho }^0=10$. Our experiments demonstrate that FedACADMM is a robust approach for both homogeneous and heterogeneous data distributions. Furthermore, our results indicate that FedACADMM performs well within a limited number of communication rounds for different datasets, and is not highly sensitive to the choice of initial variable parameters ${\rho }^0$. FedACADMM $(\rho >0)$ has better performance when the initial value is ${\rho }^0={10}^{-2}$.

Although it is not as good as ${\rho }^0={10}^{-3}$ in most cases, its performance is very stable without large deviations, and it maintains robust convergence. Therefore, we chose ${\rho }^0={10}^{-2}$ as the initial penalty parameter value for the comparison experiment with the baselines. When the initial penalty parameter value is ${\rho }^0={10}^{-2}$, the global model can effectively integrate local data and achieve robust and fast updates. It maintains a nice balance between $∥w_i^t-{\theta }^t∥$ and $∥{\theta }^t-{\theta }^{t-1}∥$. Figure 2d shows that ${\rho }^0={10}^{-3}$ stalls, which is why we did not choose it as the initial penalty parameter value. The penalty parameter that is chosen will affect how quickly ADMM converges in practice.

When we choose the appropriate initial penalty parameter value, the proposed algorithm is robust and can achieve good results in both homogeneous and heterogeneous situations. In practical applications, the convergence speed of FedACADMM is dependent on the chosen penalty parameter. The actual convergence of FedACADMM is particularly sensitive to this parameter selection. For instance, when the initial penalty parameters are set to ${\rho }^0=\left\{{10}^{-3},{10}^{-2},10\right\}$, the convergence is usually very slow. However, FedACADMM displays robustness to the choice of initial penalty parameters ${\rho }^0$. When appropriate values of the initial penalty parameters are selected, the FedACADMM algorithm demonstrates strong robustness and achieves excellent model training results and fast convergence in both IID and non-IID data distribution cases.

5.3. Comparison with Other Baselines

Comparing the proposed algorithm with existing mainstream federal optimization methods is crucial for assessing model performance. However, due to the constraints of the current communication channels, reducing the communication overhead has remained a challenge for FL. Therefore, it is reasonable to anticipate that the proposed algorithm would minimize the number of communication rounds while maintaining good heterogeneity.

In the ensuing comparative experiments, we aim to contrast the performance of FedACADMM against that of FedAvg and FedProx. Specifically, we investigate their efficacy across three extensive empirical datasets, various data distribution techniques, and diverse client numbers. Our experiments generated promising results, which was really exciting.

Table 4. Comparison of the performance of the proposed FedACADMM algorithm with FedAvg and FedProx on three datasets with different data distributions between different clients. In the header of this table, I represents IID and N represents non-IID. The best performance in each column is denoted in bold.

	MNIST(I)				CIFAR-10(I)				FMNIST(I)				MNIST(N)				CIFAR-10(N)				FMNIST(N)
	100	200	500	1000	100	200	500	1000	100	200	500	1000	100	200	500	1000	100	200	500	1000	100	200	500	1000
FedACADMM	9	13	19	15	4	9	15	21	3	5	7	16	34	22	14	10	12	9	12	13	16	15	15	13
FedAvg	18	38	85	100+	13	29	80	100+	8	16	46	97	32	43	49	65	28	23	49	79	20	32	38	51
FedProx	27	55	100+	100+	19	39	94	100+	16	22	52	100+	38	51	64	79	29	23	53	84	29	34	49	61

shows the communication rounds required to accomplish the test target accuracy under various conditions. In Table 4, we report the number of communication rounds required by each of the three algorithms to attain the target accuracy for three distinct datasets, varying data distributions, and a range of client population sizes. Additionally, we provide a graphical representation of the results in Figure 3, which facilitates a more insightful interpretation of the outcomes. The statistics of communication reduction by FedACADMM over FedAvg are shown in

Table 5. Reduced communication rounds for FedACADMM compared to FedAvg in a heterogeneous experimental setup.

	100	200	500	1000
MNIST	−6.3%	48.8%	71.4%	84.6%
CIFAR-10	57.1%	60.9%	75.5%	83.5%
FMNIST	20%	53.1%	60.5%	74.5%

. It can be observed that in most circumstances, the suggested algorithm FedACADMM can require less rounds than the baselines to attain the test target accuracy. To visually communicate our experimental findings, we have chosen to present a selection of representative results as images, as shown in Figure 4. This method provides a clear and concise representation of the data. In Figure 4a–d, the experimental results of each algorithm in the FMNIST dataset and the data distribution of IID are shown with different numbers of clients. We can observe that the performances of the three algorithms are quite good under homogeneous conditions, and the test accuracy of FedACADMM is slightly better than the other two algorithms. In Figure 4i–l, the experimental results of each algorithm in the CIFAR-10 dataset and the data distribution of non-IID are shown with different numbers of clients. When the number of clients is small, the test accuracy of FedACADMM is almost the same as that of the other two algorithms, but its advantage becomes obvious as the number of clients increases; as such, we guess that it is suitable for large-scale communication. In Figure 4e–h, the experimental results of each algorithm in the FMNIST dataset and the data distribution of non-IID are shown with different numbers of clients. It will not be repeated because its experimental results are similar to those of the CIFAR-10 dataset. In Figure 5, the corresponding train loss of test accuracy in Figure 4 is shown, and it can be seen that the convergence performance of FedACADMM is equal to the other two algorithms in most heterogeneous cases, and the convergence speed can be better than the other two algorithms in homogeneous cases. In Figure 6, we selectively display the test losses for some of the experiments that are shown in Figure 4. Figure 7 displays the confusion matrix, which indicates the performance of the predicted classes. As shown, the proposed model achieves high accuracy in classifying each dataset.

Through the detailed analysis above, we can see some advantages of FedACADMM:

Communication efficiency. Under the homogeneous data distribution method, almost all experimental methods can quickly achieve the test target accuracy, and the increase in the number of clients enables the advantages of our proposed method to be reflected. However, a homogeneous experimental setup is not informative for real-life situations. On the other hand, in a heterogeneous experimental environment, the proposed method gradually exhibits robustness with the complexity of the environment. Our proposed method successfully handles the client drift problem in diverse environments and is optimized for the local training problem.

Data Heterogeneity. Across all range of values tried, we observe that the proposed method achieves test target accuracy almost faster than baselines in both homogeneous and heterogeneous experimental settings. As the number of customers increases, the performance gap between FedACADMM and baseline gradually widens. Moreover, client sampling has a relatively small impact on the proposed method when compared to baseline experiments. With no need to manually adjust the number of local epochs, our method solves the problem of statistical heterogeneity by limiting the local updates to be more similar to the global model.

5.4. Global Collecting Step Size

As we note in Section 5.2, although ${\rho }^0={10}^{-3}$ is desirable in most cases, it is not satisfactory for FMNIST in the 1000-client and non-IID data distribution settings. In subsequent experiments, we study the convergence of the model for tuning the server cloth length under a data distribution of 1000 clients and non-IID data distributions. The reduction in data heterogeneity and random algorithm turbulence that can affect experimental results can be achieved by the global collecting step size $\eta$, as demonstrated in (11).

So we fix ${\rho }^0={10}^{-3}$ and alter $\eta \in \{0.1,0.2,0.4,0.5,0.6,0.8,1\}$. As shown in Figure 8a, $\eta \in (0.4,0.6)$ gives better test accuracy and robust convergence with ${\rho }^0={10}^{-3}$ for FMNIST in the 1000-client and non-IID data distributions setting. When the value of initial variable parameter ${\rho }^0$ is small, reducing $\eta$ synchronously to a rational range promotes better convergence. To further validate our assumption, we conduct experiments with the condition ${\rho }^0={10}^{-2}$, which was excellent in the previous experiment of changing ${\rho }^0$. It can be concluded from Figure 8b that $\eta =1$ is the best choice at ${\rho }^0={10}^{-2}$, and the test accuracy quickly reaches the optimal value and the convergence is robust and stable.

Optimizing the global collecting step size $\eta$ is critical for mitigating data heterogeneity and random algorithmic turbulence that could impede experimental outcomes. An appropriately selected global collecting step size $\eta$ ensures efficient information exchange between clients and the server, leading to improved aggregation at the server. An effective combination of $\eta$ and ${\rho }^0$ can make the algorithm experiment more effective.

6. Conclusions

In this paper, we discuss a practical scenario for federated learning that is characterized by a large number of clients with heterogeneous data and limited participation constraints. This scenario poses a challenge for achieving satisfactory convergence and model performance. To address this problem, we propose FedACADMM, a new federated objective optimization algorithm inspired by the adaptive consensus ADMM strategy in distributed optimization. Our proposed approach minimizes the communication cost per round, requiring storing additional variables at each client. This leads to a significant speed improvement as compared to the state-of-the-art model while reducing communication costs. We conducted extensive experiments on realistic datasets, and the results demonstrate that the algorithm is suitable for joint learning scenarios involving a large number of clients with heterogeneous data and limited participation constraints. Furthermore, the proposed algorithm exhibits good robustness and resilience and can achieve the target accuracy faster than the original methods while improving communication efficiency.

Federated learning has gained significant attention due to its privacy-preserving effect, achieved by uploading only the model without the raw data, and it is a rapidly growing research area. Recently, there has been increasing interest in characterizing the convergence rate of both centralized and distributed implementations of ADMM. In optimizing communication efficiency, the trade-off between reduced communication and accuracy is a critical focus of federated learning. The two most commonly studied solutions in federated optimization are bulk synchronous approaches and asynchronous approaches, where the delay is assumed to be bounded. In this paper, we only considered the bulk synchronous approach. However, in the future, we plan to consider more realistic device-centric communication schemes where each device can decide to communicate with the central server at its convenience, i.e., the asynchronous update approach.