Towards Efficient Federated Learning: Layer-Wise Pruning-Quantization Scheme and Coding Design

Abstract

As a promising distributed learning paradigm, federated learning (FL) faces the challenge of communication–computation bottlenecks in practical deployments. In this work, we mainly focus on the pruning, quantization, and coding of FL. By adopting a layer-wise operation, we propose an explicit and universal scheme: FedLP-Q (federated learning with layer-wise pruning-quantization). Pruning strategies for homogeneity/heterogeneity scenarios, the stochastic quantization rule, and the corresponding coding scheme were developed. Both theoretical and experimental evaluations suggest that FedLP-Q improves the system efficiency of communication and computation with controllable performance degradation. The key novelty of FedLP-Q is that it serves as a joint pruning-quantization FL framework with layer-wise processing and can easily be applied in practical FL systems.

1. Introduction

1.1. Background and Motivation

With the advent of the big data era, data privacy and system efficiency have attracted great attention, which also brings unprecedented challenges to the field of artificial intelligence (AI). As the first attempt to solve the problem of data silos, federated learning (FL) has now emerged as an important and promising branch of distributed machine learning [1]. Without sharing private data with other clients, FL requires only the interaction of local model parameters for cooperative learning, which significantly improves the security and efficiency of intelligent distributed systems.

The employment of AI in communication networks is already considered as a core feature of 6G systems [2]. In particular, FL has shown its strong potential in combination with deep neural networks [3,4]. As a matter of fact, FL not only naturally adapts to different structures of multi-user networks with distributed data [5], but also enables intelligent collaboration for multi-agent systems [6]. In addition, as a privacy-preserving mechanism that permits unified model training on distributed data from multiple clients, FL can be adopted in numerous fields, such as finance, healthcare, AIoT, smart cities, etc. [7].

The development of this distributed learning paradigm faces multiple challenges. In FL scenarios, clients have greater control over distributed data and local devices. Thus, there exists significant heterogeneity in terms of both data and devices. The former causes a non-iid data distribution, which hinders the convergence and the performance of the global model [8]. The heterogeneity of local devices and settings also leads to strict restrictions on the local computation and transmission, resulting in the straggler effects, where the systems have to wait for the slowest client [9]. In addition, in some mobile applications with numerous clients, the bandwidth resources are more limited and the communication costs for parameter sharing are much higher [10]. As a result, communication–computation efficiency becomes one of the most significant challenges in the practical deployment of FL systems [11]. An intuitive idea is to deploy models of different complexities on each client, which is called model heterogeneity. However, model heterogeneity violates the original assumption of FL, where the global model is supposed to be identical to all clients. Therefore, the designs of efficient heterogeneous FL systems are still in the early stage.

1.2. Related Work

In order to handle these issues related to system efficiency, two basic solutions have been considered: model pruning and parameter quantization. These were first introduced in centralized machine learning, aiming to accelerate the training process and compress the deep learning models [12]. These two techniques are broadly combined to slim the model and compress the data volume of representation. Inspired by the above two concepts, similar mechanisms have been transferred into the context of FL, which are easy to implement based on the existing FL frameworks. Specifically, pruning performed before local training or uploading, and quantization conducted before model aggregation.

The pruning strategies of FL have been broadly studied in the literature. Caldas et al. proposed the first federated pruning framework in [13], where the global model is compressed into sub-models for local clients. Based on this scheme, the pruning settings and the spectrum allocation for wireless FL were investigated in [14,15]. These methods use random width-wise pruning and have to consider different pruning rules for the inner layers with different structures. These schemes were also challenged in [16], where the authors doubted the rationality of randomly dropping the inner layer parameters. Furthermore, several new pruning mechanisms were also developed. For example, Horvath et al. pointed out that the order of the parameters matters in pruning aggregation and should be taken into consideration [17]. Depth-wise schemes based on blocks and layers were also discussed in [18,19], which showed that model compression in FL should affects the functionality of different parts in local models. The FedLP schemes in [19] specifically provided common pruning solutions for model heterogeneity cases, which extends the FL scenarios. These recent studies suggested that the pruning mechanisms should be designed specially due to the characteristic aggregation operation in FL.

As for quantization, it is commonly treated as an independent procedure from pruning. QSGD, a classic quantization scheme proposed in [20], has been integrated into FL systems [21,22]. Some extensions and variants for communication-efficient scenarios were also developed in [23,24]. The theoretical analysis was studied in [25], and some universal coding schemes were adopted in [26]. In addition, Prakash et al. investigated optimized FL by combining pruning and quantization [27]. However, specific coding schemes that fit the aforementioned pruning and quantization are essentially required. As combined technologies for model compression, pruning-quantization-coding can be jointly and compatibly designed to improve the efficiency for practical FL systems.

1.3. Contribution and Organization

Inspired by the above studies, we mainly consider a compressed FL scheme with joint layer-wise model pruning and parameter quantization in this work, which is an extension of [19]. The main contributions of this work can be summarized as follows:

•

We propose FedLP-Q (the codes in this work are available at https://github.com/Zhuzzq/FedLP/tree/main/FedLP-Q), a joint pruning-quantization FL framework employing the layer-wise approach. FedLP-Q provides a simple but efficient paradigm to mitigate the communication loads and computation complexity in FL systems;

•

We develop two distinct FedLP-Q schemes for homogeneous and heterogeneous cases. The corresponding pruning strategies, quantization rules, and the coding designs are presented in detail. The theoretical analysis also shows the strengths of the coding scheme for FedLP-Q;

•

We carry out several experiments to evaluate the performance of FedLP-Q. The outcomes suggest that this layer-wise pruning-quantization mechanism significantly improves the system efficiency with controllable performance loss.

The remainder of this article is organized as follows. In Section 2, we introduce the preliminaries and the basic concept of layer-wise pruning-quantization. In Section 3, we illustrate the specific design of the proposed approach and its theoretical analysis. The detailed experimental results and more discussions are presented in Section 4. Finally, in Section 5, we conclude and show several potential directions for this work.

2. Preliminaries and FedLP-Q Framework

In this section, we introduce some preliminaries and the framework of FedLP-Q. Before that, for explicity, we list some basic notations in

Table 1. Basic Notations.

Notations	Description
N	The number of distributed clients.
$E_l$	Local training epoch.
$\left\{{\mathcal{D}}_1,\cdots ,{\mathcal{D}}_N\right\}$	Distributed local datasets.
$P_t$	The set of participation clients at t-th global epoch.
K	The number of participation clients.
$\overline{\mathit{\theta }}$	The global model.
$\left\{{\mathit{\theta }}_1,\cdots ,{\mathit{\theta }}_N\right\}$	Local models.
$\left\{{\omega }_1,\cdots ,{\omega }_N\right\}$	Aggregation weights for each client.
${\mathit{g}}_{k,t}$	The accumulated gradient of client k in t-th global epoch.

.

2.1. Basic FL

In a typical FL system with N distributed clients, each client k possesses its own dataset ${\mathcal{D}}_k$ to train its local model ${\mathit{\theta }}_k$. To preserve the privacy of personal data, FL conducts local training and periodic model aggregation. Clients cooperate to update the global model $\overline{\mathit{\theta }}$ via parameter sharing, which improves the communication efficiency. As a standard rule, FedAvg [1] selects K clients as a temporary active group $P_t$ and aggregates their local gradients to update the global model. At each global round t, participator clients train their local models for $E_l$ epochs and upload the local gradients to the server. The global model is updated by model aggregation as follows:

$${\overline{\mathit{\theta }}}_{t+1}\leftarrow {\overline{\mathit{\theta }}}_t-\sum _{k\in P_t}\frac{{\omega }_k}{{\displaystyle \sum _{m\in P_t}}{\omega }_m}·{\eta }_k{\mathit{g}}_{k,t},$$

(1)

where ${\mathit{g}}_{k,t}$ is the accumulated gradients of client k after $E_l$ epoch local training. ${\omega }_k=\frac{|{\mathcal{D}}_k|}{\sum |{\mathcal{D}}_m|}$ is the aggregation weights and ${\eta }_k$ is the learning rate of each client.

2.2. An Overview of the FedLP-Q Framework

As discussed above, the aggregation mechanism of FL makes it possible to conduct layer-wise processes for local models. Furthermore, FL with layer-wise pruning has shown its strengths in several aspects including stability, convergence, and communication–computation efficiency [19]. We generalize such schemes by adding the quantization for layer-wise pruning and formulate the FL framework with layer-wise pruning-quantization, abbreviated as FedLP-Q.

Consider a full model with L layers, $\mathit{\theta }:=\left[{\mathit{\theta }}^1,{\mathit{\theta }}^2,\cdots ,{\mathit{\theta }}^L\right]$. After a round of local training on client k, its accumulated gradients can be represented by:

$${\tilde{\mathit{g}}}_k=\left[{\mathit{g}}_k^l\right],l\in {\mathcal{L}}_k,$$

(2)

where ${\mathcal{L}}_k$ is the index set of the preserved layers and is determined by the specific layer-wise pruning strategies. The pruned accumulated gradients of each layer are further quantized into $Q\left({\mathit{g}}_k^l\right)$, where $Q(·)$ denotes the layer-wise quantization function. Hence, the layer-wise aggregation rule shall be modified as:

$${\overline{\mathit{\theta }}}_{t+1}^l\leftarrow {\overline{\mathit{\theta }}}_t^l-\sum _{k\in P_t}\frac{{\mathbb{1}}_k^l·{\omega }_k{\eta }_k}{{\sum }_{m\in P_t}{\mathbb{1}}_m^l·{\omega }_m}Q\left({\mathit{g}}_{k,t}^l\right),$$

(3)

where ${\mathbb{1}}_k^l$ is the probabilistic indicator function with value $\{0,1\}$ to represent whether the l-th layer is preserved for the local model on client k. The examples of the indicators will be presented in the following section.

Figure 1 shows the basic frameworks of FedLP-Q for homogeneous and heterogeneous local model scenarios. In the homogeneity scheme, all clients possess the same full layers of the global model. In the heterogeneity scheme, clients initialize their local pruned models with different sequential layers. During the aggregation rounds, clients prune their local models according to layer-wise pruning. Then, the sub-models are quantized into lower bit rates and encoded for model uploading. The parameter server decodes and aggregates the quantized gradients. Similarly, the updated global model will be encoded and transmitted to each local client. In FedLP-Q, since each layer of the model is the basic unit, the pruning and quantization processes can be explicit and model-agnostic. For the above reasons, layer-wise pruning-quantization schemes require no previous knowledge of the model and can easily be deployed in FL systems.

3. FedLP-Q: Quantization-Coding Scheme for Layer-Wise Pruning

In this section, we will introduce the proposed layer-wise pruning-quantization approach and the coding scheme in detail. The convergence and the coding performance of FedLP-Q will also be analyzed.

3.1. Pruning Phase

To develop layer-wise pruning for FL, we shall consider the homogeneous and heterogeneous local models. Similar to [19], we adopt the probabilistic layer-wise pruning strategy for homogeneity cases. For heterogeneity cases, we assign the pruned sub-models with different sequential layers to the clients.

3.1.1. Homogeneity Case

As shown in Figure 1a, all clients possess the same full model. While finishing the local training, clients carry out layer-wise pruning for the accumulated gradients. Probabilistic layer-wise pruning proceeds as follows: before uploading the accumulated gradients for aggregation, each client k preserves the gradients of the l-th layer with probability $p_k^l$, termed as the layer-preserving rate (LPR). In other words, the indicators in (3) satisfies:

$${\mathbb{1}}_k^l=\left\{\begin{array}{ccc}1\hfill & \hfill \mathrm{with}\mathrm{probability}& p_k^l,\hfill \\ 0\hfill & \hfill \mathrm{with}\mathrm{probability}& 1-p_k^l.\hfill \end{array}\right.$$

(4)

The pruned accumulated gradients ${\tilde{\mathit{g}}}_k$ in (2) for homogeneity cases can be formulated according to $\left\{{\mathbb{1}}_k^l\right\}$.

3.1.2. Heterogeneity Case

The major challenge of FL with heterogeneous local models is to design the rules for model assignment and aggregation. Fortunately, the layer-wise mechanism of FedLP-Q gives an explicit solution to such problems. That is, each client initializes the local model of different layers. Let $L_k$ denote the layer count (LC). As shown in Figure 1b, its local model shall be initially pruned into ${\tilde{\mathit{\theta }}}_k=\left[{\mathit{\theta }}_k^1,\cdots ,{\mathit{\theta }}_k^{L_k},\left({\mathit{\theta }}_k^O\right)\right]$, where ${\mathit{\theta }}_k^O$ (if it exists) is the personalized fully connected (FC) layer to fit the output dimensions and will not be uploaded for aggregation. Then, the pruned accumulated gradients of client k in heterogeneity settings can be expressed as:

$${\tilde{\mathit{g}}}_k=\left[{\mathit{g}}_k^1,\cdots ,{\mathit{g}}_k^{L_k}\right].$$

(5)

and the layer indicators $\left\{{\mathbb{1}}_k^l\right\}$ can be determined.

In this case, the computation complexity for local training is notably reduced since client k only needs to train a sub-model with $L_k$ layers. As a result, the straggler clients are able to contribute to the global model by training and transmitting small sub-models. This scheme considers the imbalance of the local devices and their computation–communication capabilities, which is more practical in real scenarios.

3.2. Quantization and Coding Phase

Based on layer-wise pruning, we further formulate the corresponding quantization settings and the coding scheme.

3.2.1. Quantization Scheme

For each preserved layer, FedLP-Q conducts the quantization method with equal intervals, which is similar to the quantized-SGD (QSGD) in [20]. Layer-wise quantization after pruning proceeds as follows: for a preserved layer l on client k and the preset quantization bit b, the quantized aggregated gradient $Q({\mathit{g}}_k^l;b)$ is defined by:

$$Q({\mathit{g}}_k^l;b)=\frac{{∥{\mathit{g}}_k^l∥}_2}{2^b}·\mathrm{sgn}\left({\mathit{g}}_k^l\right)\odot \mathrm{seg}\left({\mathit{g}}_k^l;b\right),$$

(6)

where ⊙ is the element-wise product, $\mathrm{sgn}(·)$ is the sign function, and $\mathrm{seg}\left({\mathit{g}}_k^l;b\right)$ is the interval segmentation function. Specifically, for the full gradients ${\mathit{g}}_k^l$ of the l-th layer, we firstly divide its ${\ell }_2$-norm into $2^b$ equal intervals, which takes b bits. Then, each gradient in the layer follows a stochastic quantization: for $g_k^{(l,i)}$, if it locates at the s-th interval, i.e., $|g_k^{(l,i)}|/\parallel {\mathit{g}}_k^l{\parallel }_2\in [\frac{s}{2^b},\frac{s+1}{2^b}]$, the gradient $g_k^{(l,i)}$ will be quantized as the values of two sides according to the distances from the two endpoints of the interval. The quantization can be expressed as:

$$\mathrm{seg}\left(g_k^{(l,i)};b\right)=\left\{\begin{array}{cc}s+1\hfill & \hfill \mathrm{with}\mathrm{probability}{p}_Q^{(l,i)}(s;b),\\ s\hfill & \hfill \mathrm{otherwise}.\end{array}\right.$$

(7)

Therein, the stochastic quantization probability is defined as:

$$p_Q^{(l,i)}(s;b)=\frac{|g_k^{(l,i)}|}{\parallel {\mathit{g}}_k^l{\parallel }_2}·2^b-s,$$

(8)

which is the ratio between the distance to the left endpoint and the interval length.

3.2.2. Coding Design

Given the quantization bit b and the layer 2-norm $\parallel {\mathit{g}}_k^l{\parallel }_2$, the quantized value of each gradient can be handled as the index of the interval. Then, the coding of the quantized gradients can be considered as the representation of the integers. Therefore, we adopt a classic solution: recursive Elias coding, which is also known as Elias omega coding  [28]. The core idea of Elias omega coding is to recursively encode the prefix of the binary representation. Hence, the code length of any positive integer N follows $\left|\mathrm{Elias}\right(N\left)\right|=1+logN+loglogN+\cdots =1+(1+o(1\left)\right)logN$. Then, we shall consider the simple encoding scheme for layer-wise pruning-quantization. The full coding scheme for FedLP-Q proceeds as follows: For each preserved layer l, the layer 2-norm will firstly be encoded into 32 bits (float). The quantization setting b (integer) will be attached using Elias omega coding. Then, for each gradient, the index of interval ${\mathrm{seg}}_k^l\left(i\right)$ for stochastic quantization will be sequentially encoded according to the Elias omega coding. The sign of each gradient will additionally take 1 bit (a boolean variable standing for positive/negative). The decoding scheme is also simple: The decoder firstly reads off the 32 bits to obtain the layer 2-norm $\parallel {\mathit{g}}_k^l{\parallel }_2$ and decodes the Elias codes of b. Then, for each gradient, its interval index will be reconstructed according to Elias omega decoding, while the sign will be decoded by the last bit.

After layer-wise pruning-quantization, the layer gradients can be represented by a 4-element tuple $\left\{\parallel {\mathit{g}}_k^l{\parallel }_2,b,\mathrm{seg}({\mathit{g}}_k^l;b),\mathrm{sgn}\left({\mathit{g}}_k^l\right)\right\}$. Therefore, based on Elias omega coding, the code structure of each layer with $M_l$ parameters is shown in Figure 2. Furthermore, the corresponding encoding and decoding procedures can be developed as shown in Algorithm 1 and Algorithm 2, respectively.

Algorithm 1 Encoder for layer l
Input:$\left\{\parallel {\mathit{g}}_k^l{\parallel }_2,b,\mathrm{seg}({\mathit{g}}_k^l;b),\mathrm{sgn}\left({\mathit{g}}_k^l\right)\right\}$, $M_l$;
1: ${\mathcal{C}}_l\leftarrow \mathtt{Concat}\left(\mathtt{Float}\mathtt{32}\left(\parallel {\mathit{g}}_k^l{\parallel }_2\right),\mathtt{Elias}\_\mathtt{Encoder}\left(b\right)\right)$;	▹encode the header
2: for  $i\leftarrow 1\mathbf{to}{M}_l$  do
3:    ${\mathcal{C}}_l\leftarrow \mathtt{Concat}\left({\mathcal{C}}_l,\mathtt{Elias}\_\mathtt{Encoder}\left(\mathrm{seg}({\mathit{g}}_k^{(l,i)};b)\right)\right)$;	▹ encode the interval index
4:    ${\mathcal{C}}_l\leftarrow \mathtt{Concat}\left({\mathcal{C}}_l,\mathtt{Binary}\left(\mathrm{sgn}({\mathit{g}}_k^{(l,i)};b)\right)\right)$;	▹ encode the gradient sign
5: end for
Output: encoded message ${\mathcal{C}}_l$.

Algorithm 2 Decoder for layer l
Input: received message ${\mathcal{C}}_l$; Initialization: ${\mathrm{seg}}^l\leftarrow \left[\right]$, ${\mathrm{sgn}}^l\leftarrow \left[\right]$, $i\leftarrow 0$;
1: $G_l\leftarrow \mathtt{Float}\mathtt{32}\_\mathtt{Decoder}\left({\mathcal{C}}_l\right)$;	▹decode the gradient norm
2:  ${\mathcal{C}}_l>>32$;
3: $(b,l_E)\leftarrow \mathtt{Elias}\_\mathtt{Decoder}\left({\mathcal{C}}_l\right)$;	▹decode the quantization bit
4: ${\mathcal{C}}_l>>l_E$;
5: while  ${\mathcal{C}}_l$  do
6:     $({\mathrm{seg}}_i^l,l_E)\leftarrow \mathtt{Elias}\_\mathtt{Decoder}\left({\mathcal{C}}_l\right)$;	▹ decode the interval index
7:     ${\mathcal{C}}_l>>l_E$;
8:     $({\mathrm{sgn}}_i^l,l_E)\leftarrow {\mathcal{C}}_l$;	▹ decode the gradient sign
9:     $i\leftarrow i+1$; ${\mathcal{C}}_l>>1$;
10: end while
Output: decoded elements $\left\{G_l,b,{\mathrm{seg}}^l,{\mathrm{sgn}}^l\right\}$.

3.2.3. Algorithm Formulation

Finally, we formulate the procedures of FedLP-Q as in Algorithm 3. The main changes from the original FL include (1) layer-wise pruning (for both the homogeneity and heterogeneity case ) for the local gradients is conducted in Line 5; (2) stochastic quantization and the corresponding encoding/decoding are performed in Lines 6 and 7; and (3) Line 9 introduces the layer-wise aggregation for the proposed pruning-quantization schemes.

Algorithm 3 FedLP-Q
Initialization: local models, pruning configures, etc.
1: for $t\leftarrow 1\mathbf{to}\max \_\mathrm{epoch}$ do
2:     for participator client k in parallel do	▹ client side
3:         Update ${\mathit{\theta }}_{k,t}\leftarrow \mathtt{Local}\_\mathtt{Train}({\mathit{\theta }}_{k,t-1};{\mathcal{D}}_k)$;
4:         Calculate the accumulated gradients ${\mathit{g}}_{k,t}$;
5:         Conduct layer-wise pruning: ${\tilde{\mathit{g}}}_{k,t}$;
6:         Quantize the local gradients: $Q\left({\tilde{\mathit{g}}}_{k,t}\right)$;
7:         Carry out Algorithm 1 to encode the quantized gradients;
8:         Upload local message to parameter server;
9:     end for
10:     Carry out Algorithm 2 to decode the received messages;	▹ server side
11:     Aggregate each layer ${\overline{\mathit{\theta }}}_t^l$ by (3);
12:     Download the global model: ${\mathit{\theta }}_{k,t}\leftarrow {\overline{\mathit{\theta }}}_t$;	▹ client side
13: end for
Output: global model: ${\overline{\mathit{\theta }}}_t$.

3.3. Theoretical Analysis

Then, we theoretically analyzed the convergence and coding performance of FedLP-Q.

3.3.1. Impacts on Convergence

Since FedLP-Q modifies the aggregation process of the original FL, the main impacts on convergence come from layer gradient pruning and quantization. Assuming that local training is independent from the pruning operations, the following proposition indicates the convergence result of FedLP-Q.

Proposition 1.

Consider a fairness case where ${\omega }_k=1/K$ and $p_k^l=p$ for all $k=1,\cdots ,K$. FedLP-Q leads to ${(1-p)}^K$ convergence rate decay compared to the original FL scheme, i.e., the expectation of the aggregated gradient in FedLP-Q satisfies:

$$\mathbb{E}\left[{\widehat{\mathit{g}}}^l|{\overline{\mathit{g}}}^l\right]=\left[1-{(1-p)}^K\right]·{\overline{\mathit{g}}}^l.$$

(9)

Therein,

$${\overline{\mathit{g}}}^l=\frac{1}{K}\sum _k{\mathit{g}}_k^l,{\widehat{\mathit{g}}}^l=\sum _k\frac{{\mathbb{1}}_k^l}{{\sum }_m{\mathbb{1}}_m^l}Q\left({\mathit{g}}_k^l\right)$$

(10)

denote the aggregated gradients in the original FL and FedLP-Q, respectively.

Proof. 

For simplicity, rewrite the client index in $P_t$ as $\{1,2,\cdots ,K\}$. According to the layer-wise aggregation in (3), the expectation of the pruned-quantized gradients can be obtained by:

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\widehat{\mathit{g}}}^l|{\overline{\mathit{g}}}^l\right]& =\mathbb{E}\left[\sum _{k=1}^K\frac{{\mathbb{1}}_k^l}{{\sum }_m{\mathbb{1}}_m^l}Q\left({\mathit{g}}_k^l\right)|{\overline{\mathit{g}}}^l\right]=\sum _{k=1}^K\mathbb{E}\left[\frac{{\mathbb{1}}_k^l}{{\sum }_m{\mathbb{1}}_m^l}Q\left({\mathit{g}}_k^l\right)|{\overline{\mathit{g}}}^l\right]\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \stackrel{➀}{=}\sum _{k=1}^K\mathbb{E}\left\{\mathbb{E}\left[\frac{{\mathbb{1}}_k^l}{{\sum }_m{\mathbb{1}}_m^l}Q\left({\mathit{g}}_k^l\right)|{\overline{\mathit{g}}}^l,{\mathbb{1}}_k^l\right]|{\overline{\mathit{g}}}^l\right\}\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & =\sum _{k=1}^{K}p·\mathbb{E}\left[\frac{1}{1+{\sum }_{m\ne k}{\mathbb{1}}_m^l}·Q\left({\mathit{g}}_k^l\right)|{\overline{\mathit{g}}}^l\right]\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & =\sum _{k=1}^{K}p\left[\sum _{m=1}^K\frac{1}{m}\left(\genfrac{}{}{0pt}{}{K-1}{m-1}\right)p^{m-1}{(1-p)}^{K-m}\right]\mathbb{E}\left[Q\left({\mathit{g}}_k^l\right)|{\overline{\mathit{g}}}^l\right]\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & \stackrel{➁}{=}\sum _{k=1}^K\left[\sum _{m=1}^K\frac{1}{K}\left(\genfrac{}{}{0pt}{}{K}{m}\right)p^m{(1-p)}^{K-m}\right]\mathbb{E}\left[Q\left({\mathit{g}}_k^l\right)|{\overline{\mathit{g}}}^l\right]\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & =\left[1-{(1-p)}^K\right]\mathbb{E}\left[\sum _{k=1}^K\frac{Q\left({\mathit{g}}_k^l\right)}{K}|{\overline{\mathit{g}}}^l\right]\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & \stackrel{➂}{=}\left[1-{(1-p)}^K\right]·{\overline{\mathit{g}}}^l,\hfill \end{array}$$

(17)

where ➀ is the result of equation $\mathbb{E}\left[X\right|Y]=\mathbb{E}\left\{\mathbb{E}\left[X\right|Y,Z\left]\right|Y\right\}$, ➁ holds because $\frac{1}{m}\left(\genfrac{}{}{0pt}{}{K-1}{m-1}\right)=\frac{(K-1)!}{(K-m)!m!}=\frac{1}{K}\left(\genfrac{}{}{0pt}{}{K}{m}\right)$, and ➂ is the unbiased quantization property of QSGD-based quantizers [22,25]. Then, the proposition is proved. □

Remark 1.

(9) shows that the aggregated gradients of FedLP-Q decreases compared to the original FL schemes, which implies ${(1-p)}^K$ convergence rate decay. In addition, the conclusion of Proposition 1 can be directly adopted in the assumptions of model pruning/quantization (e.g., [22,24]) and the specific convergence theorems can be further modified.

According to the proposition, such impacts on convergence can be mitigated by increasing either the layer preserving rate p or the number of participator clients K.

3.3.2. Coding Performance

Based on the stochastic quantization method and the corresponding coding scheme, we obtained the following proposition, which highlights the communication costs of FedLP-Q.

Proposition 2

(Code length). By employing layer-wise pruning-quantization, the average bit number to communicate ${\mathit{g}}_k^l$ for client k is upper-bounded by:

$$\widehat{\mathcal{C}}\left({\mathit{g}}_k^l;b\right)=33+\left(1+o\left(1\right)\right)logb+M_l\left[2+\left(1+o\left(1\right)\right)b\right].$$

(18)

Proof. 

The proposition is an intuitive result of the proposed quantization coding scheme. As discussed above, the code length of an integer N using Elias omega coding satisfies $\left|\mathrm{Elias}\right(N\left)\right|=1+logN+loglogN+\cdots =1+(1+o(1\left)\right)logN$. Consider the code blocks of the l-th layer as shown in Figure 2, the code length can be divided into three parts: (1) The first block to represent the layer 2-norm $\parallel {\mathit{g}}_k^l{\parallel }_2$ takes 32 bits; (2) the Elias codes of the quantization level b takes $1+(1+o(1\left)\right)logb$ bits; and (3) after stochastic quantization, the interval index of each parameter is an integer less than $2^b$, which takes $1+(1+o(1\left)\right)b$ bits, and its sign takes one more bit. Hence, the total code length can be upper bounded as shown in (18). □

Remark 2.

According to [20], the original coding scheme proposed in QSGD for the l-th layer leads to the average code length asymptotically bounded by:

$$32+\left[3+\left(1.5+o\left(1\right)\right)log\frac{2(S^2+M_l)}{S(S+\sqrt{M_l})}\right]S(S+\sqrt{M_l}),$$

(19)

where $S=2^b$ is the count of the quantization intervals. Comparing the two upper bounds of encoding the layer-wise gradients in (18) and (19), one can find that the proposed coding scheme for FedLP-Q has a shorter code length than that in the original QSGD when the number of parameters is not extremely large.

Specifically, the Elias coding approach for QSGD considers the positional coding of the non-zero gradient to fit the cases with sparse and numerous parameters. On the contrary, in FedLP-Q, the gradients of each layer are handled independently. Therefore, the sparsity and the element amounts are not enough to show the advantages of the Elias coding scheme in QSGD. The experimental results in the next section also indicate that the quantization coding for FedLP-Q can be simple but more efficient. Overall, by employing layer-wise pruning and quantization, FedLP-Q improves the communication–computation efficiency in FL, and also simplifies the code design for model uploading.

Furthermore, we discuss the code efficiency. After layer-wise pruning-quantization, the gradients of $Q\left({\mathit{g}}_k^l\right)$ can be represented by a set of interval indexes. Then, the entropy rate of the layer gradients can be defined as follows:

Definition 1.

After the b-bit quantization, the entropy rate of the layer gradients ${\mathit{g}}_k^l$ is defined as:

$$\mathcal{H}\left(Q\left({\mathit{g}}_k^l\right);b\right):=\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^n\mathcal{H}\left({\mathrm{seg}}_k^l\left(i\right)\right);$$

(20)

where $\mathrm{seg}(·)$ is the quantization index. Consider the independence of each gradient and the empirical distribution, the above entropy rate can be approximated by:

$$\widehat{\mathcal{H}}\left(Q\left({\mathit{g}}_k^l\right);b\right)\approx \widehat{\mathcal{H}}\left({\mathrm{seg}}_k^l\right):=-\sum _{s\in \left\{{\mathrm{seg}}_k^l\right\}}f\left(s\right)log\left(f\left(s\right)\right),$$

(21)

where $f\left(s\right)=\frac{\mathtt{count}\left(s\right)}{M_l}$ is the proportion of index s in ${\mathrm{seg}}_k^l$.

Then, the code efficiency can be investigated by comparing the per-gradient code length and the entropy rate. According to Proposition 2, the per-gradient code length can be expressed asymptotically by:

$$\frac{\widehat{\mathcal{C}}\left({\mathit{g}}_k^l;b\right)}{M_l}\approx 2+\left(1+o\left(1\right)\right)b.$$

(22)

Hence, the aim is to make the asymptotic per-gradient code length as close as possible to the entropy rate, i.e., $\frac{\widehat{\mathcal{C}}\left({\mathit{g}}_k^l;b\right)}{M_l}\to \widehat{\mathcal{H}}\left(Q\left({\mathit{g}}_k^l\right);b\right)$. Moreover, consider the whole procedure of FedLP-Q, the equivalent per-gradient code length $p_k^l\frac{\widehat{\mathcal{C}}\left({\mathit{g}}_k^l;b\right)}{M_l}$ can be even smaller than the entropy rate $\widehat{\mathcal{H}}\left(Q\left({\mathit{g}}_k^l\right);b\right)$, where $p_k^l$ is the preserving proportion of the gradients of the l-th layer. In the next section, we will show that the proposed FedLP-Q can achieve the above goals through experiments.

4. Experimental Results

In this section, we describe several experiments to evaluate the performance of the FedLP-Q schemes.

We carried out the experiments on an image classification FL task on CIFAR-10 with 100 clients and a 0.1 participation rate. Specifically, the parameter server randomly selected 10 clients for aggregation in every global round. The number of the local training epoch was fixed at 5. We adopted the global model as a CNN-based neural network with six Conv layers and two FC layers. Batch normalization and maxpooling were also conducted following each Conv layer. As for the two proposed schemes, we adopted the consistent LPR p for the homogeneous case, abbreviated as FedLP-Q_Homo(p). For the heterogeneous case, we separated the Conv layers into five ordered sequences with LC from 1 to 5. These five sub-models were assigned to clients according to LC distributions. FedLP-Q_Hetero(l) means that the model with l LC was assigned with the highest probability and the parameter ’u’ represents the uniform assignment.

We evaluated the performance of the proposed FedLP-Q schemes under both iid and Dirichlet non-iid data ($\alpha$ = 1) settings. As shown in Figure 3, the FedLP-Q schemes reached a similar accuracy and convergence as the original FedAvg. In particular, the homogeneous case with high quantization bits (b = 10) led to nearly the same test accuracy curves as that of FedAvg, but significantly reduced the communication traffic. Small quantization bits and the non-iid data settings led to a lower accuracy and the instability of the global model. Therefore, there exists redundancy in the transmitted gradients and the quantization representation. By setting different FedLP-Q configures, FL systems can reduce the requirement of the communication rate and computation capability, which enables more devices to participate in the FL tasks.

The detailed numerical results for the test accuracy and the communication–computation efficiency are listed in

Table 2. Comparisons on accuracy, communication, and computation saving.

Schemes	Test Accuracy % (iid/Non-iid)			#Param ↘	MFLOPs ↘
	No Quant	b = 10	b = 8
FedAvg	78.8/75.3	76.2/74.9	71.7/68.5	/	/
FedLP-Q(0.2)	73.4/68.2	71.5/66.8	56.6/50.4	40.0%	/
FedLP-Q(0.5)	75.5/73.5	75.4/72.5	66.7/62.0	25.0%	/
FedLP-Q(0.8)	78.9/77.3	78.3/76.2	71.3/66.9	10.0%	/
FedLP-Q(1)	64.4/60.3	64.1/59.2	53.6/49.4	84.6%	51.2%
FedLP-Q(3)	66.8/62.0	66.1/61.7	58.4/53.9	79.6%	32.3%
FedLP-Q(u)	70.1/66.5	68.3/65.4	60.7/55.7	71.1%	34.5%
FedLP-Q(5)	74.6/73.1	74.4/72.1	68.6/64.5	35.6%	17.2%

. The savings of communication and computation are represented by the decay ratios of the parameter count and the million floating point of operations (MFLOPs) per local model, respectively. One can find that, for homogeneity cases, FedLP-Q performs better on both data settings. The randomness caused by the probabilistic layer-wise pruning and stochastic quantization even benefit the model performance. In addition, the number of parameters can be reduced, which relieves the communication loads. For heterogeneity cases, though there exists degradation of the test accuracy, FedLP-Q can significantly reduce both the communication and computation costs with acceptable model performance. This is an important characteristic, especially for mobile scenarios where the clients’ capabilities of transmission and local computation are limited.

Figure 4 displays the model performance, communication traffic, and computation load savings of several schemes, including FedAvg, homogeneous FedLP-Q, and heterogeneous FedLP-Q, with different quantization settings. The higher vertical axis represents a higher model accuracy. The x-axis denotes the reduced communication traffic (MB) and the y-axis is the saved local computation complexity (MFLOPs). The horizontal plane implies the communication and computation efficiency. If a projection on the x-y plane lies farther from the zero point, the scheme saves more communication and computation loads. By plotting the 3D figure, the trade-offs between the model performance and system efficiency can be intuitively shown. According to this trade-off, the corresponding system design and the pruning-quantization settings can be well guided.

We next evaluated the efficiency of the coding scheme for FedLP-Q. Figure 5a shows how the code length evolves as the number of the parameters increases. We compared the expected code length of the proposed Elias coding for FedLP-Q (as in (18)), the Elias coding for QSGD (as in (19)), and the non-quantization coding (Float32). For a parameter count of less than 600 K, the Elias-based coding scheme proposed in Section 3.2.2 led to the lowest expected code length. The superiority of the QSGD coding scheme in [20] appeared for an extremely large parameter count. Notably, since we adopted a layer-wise approach, the parameter count of each layer commonly fell within the red region marked in Figure 5b. As a result, the proposed coding scheme is more suitable for FedLP-Q. Figure 5b presents the practical code length of the neural network model employed in the experiments. By implementing the Elias code, FedLP-Q encodes the accumulated gradients into a lower bit rate, which reduces the requirement for the communication rate in real-world systems.

Finally, we investigated the gap between the per-gradient length of the proposed schemes and the entropy rate of the gradients. Figure 6 displays the quantized gradient distribution of the selected clients under iid and non-iid data settings. The quantized gradient distributions of different layers have similar shapes under both iid and non-iid settings. This again demonstrates the potential and future directions of introducing layer-wise operation into FL. That is, the distribution and the entropy of each layer gradients can be modeled. Then, the communication traffic of each client can be better estimated and further coding schemes can be specifically designed.

Figure 7 shows the entropy rate $\widehat{\mathcal{H}}\left(Q\left({\mathit{g}}_k^l\right);b\right)$ and the equivalent per-gradient code length $p_k^l\frac{\widehat{\mathcal{C}}\left({\mathit{g}}_k^l;b\right)}{M_l}$ of each layer. Herein, the entropy rate of the gradients can be calculated through (21) based on the distributions shown in Figure 6. It is intuitive that the per-gradient length of the proposed schemes is close to the entropy rate of the gradients. In particular, as discussed in Section 3.3.2, with a lower LPR, the expectation of the per-gradient code length can take even fewer bits than the entropy rate, e.g., as shown by the curves of $p_k^l=0.7$. This indicates that the proposed coding scheme is efficient and can be adjustable to meet different system requirements on compression.

All the above results suggest that there exists a significant redundancy in the interaction of FL systems. Layer-wise pruning-quantization and the proposed coding design can be applied without damaging the model performance. Therefore, it can be concluded that FedLP-Q is an efficient scheme that improves the system efficiency and facilitates the deployment of FL applications by mitigating the straggler effects.

5. Conclusions

In this article, we proposed FedLP-Q, an efficient pruning-quantization approach for FL by adopting layer-wise operations. The corresponding pruning strategies, stochastic quantization, and coding scheme were formulated. Both the theoretical and experimental analyses indicated that FedLP-Q achieves a better communication–computation efficiency with controllable performance loss. This may be vital for the deployment of FL applications in mobile scenarios.

This work also opens up several potential research directions for future work, including exploring the robustness and security of layer-wise pruning-quantization, combinations with other FL structures, quantization/coding designs based on layer-gradient entropy estimation, and more practical modeling for the communication of FL clients.