CE-VFAL: A Novel Framework for Communication-Efficient Vertical Federated Adversarial Learning

1 Introduction

Federated Learning (FL) [Kairouz et al., 2021; Yang et al., 2019; McMahan et al., 2016] represents a distributed learning paradigm that empowers multiple clients to collectively train and utilize a machine learning (ML) model without sharing their data directly. Conventionally, most FL research considers the Horizontal FL (HFL) setting, where clients hold different data samples with the same feature space. In contrast, vertical FL (VFL) tackles the scenarios where clients have identical samples but disjoint feature spaces [Liu et al., 2024; Wei et al., 2022]. A typical VFL model comprises a top model maintained by a server and multiple bottom models, each residing at participating clients. Throughout the training, individual clients compute local embeddings of data features using their respective bottommodels and transmit them to the server through a communication channel. The server aggregates these embeddings to update the top model and then sends gradients back to clients to update bottom models accordingly. Therefore, recent researches focuses on studying communication efficiency, involving reducing the cost of coordination and compressing the data transmitted between parties, such as multiple client updates [Zhang et al., 2022], asynchronous coordination [Li et al., 2020], and data compression [Castiglia et al., 2022; Li et al., 2020].

Threat model. ML models have demonstrated vulnerability to adversarial attacks, carefully crafted inputs designed to induce misclassification during inference. Distributed deployment inherently exposes VFL model to additional security threats [Huang et al., 2024; Duanyi et al., 2023]. Adversarial attacks in VFL can manifest in multiple forms during inference: via third-party adversary intercepting and altering embeddings during client-server communication [Duanyi et al., 2023], or through malicious or colluding clients perturbing local features of raw data [Pang et al., 2022; Qiu et al., 2022]. The details of the threat model are presented in Appendix A.3. These diverse attacks underscore the unique security challenges in VFL systems, motivating the urgent need to address the VFL robustness problem.

Currently, much effort has been made to improve models' resistance to such perturbations in centralized training scenarios. Among these studies, the Adversarial Training (AT) scheme has demonstrated significant potential in bolstering the robustness of ML models[Tram`er et al., 2018]. AT is a min-max robust training method that minimizes the worstcase training loss at adversarially perturbed examples [Madry et al., 2017]. The deployment of AT in the FL paradigm termed Federated Adversarial Learning (FAL), has garnered attention, with a particular focus on HFL scenarios, where each participant maintains a complete copy of the model [Li et al., 2023]. However, in VFL scenarios, a single global model is partitioned and distributed among the server and clients, resulting in a different architecture for Vertical Federated Adversarial Learning (VFAL).

Observation. Due to layer-wise distributed deployment, VFAL presents unique challenges in communication efficiency. Adversarial sample generation during AT is computationally intensive, requiring sequential forward and backward propagation to calculate gradients concerning the input for iterative refinement [Madry et al., 2017]. As figure 1-left shows, VFAL using PGD-10 (Projected Gradient Descent) requires about 10 times more communication cost than regular VFL due to 10 iterations needed to generate adversarial example. Consequently, significant communication costs pose an unacceptable barrier between robustness and communication efficiency, hindering VFAL framework development.

Recognizing these challenges, we propose a novel framework named Communication-Efficient Vertical Federated Adversarial Learning (CE-VFAL) via introducing the lazy propagation principle and Zeroth Order Optimization (ZOO)[Wang et al., 2024], aimed at enhancing VFL model robustness through communication-efficiency AT: 1) By integrating lazy propagation, CE-VFAL reduces the frequency of interactions between clients and the server by confining the majority of propagation in client parties during adversarial updates (Figure 1- 1⃝); 2) CE-VFAL incorporates ZOO into the communication phase to reduce the communication load of the backward propagation by transmitting only the loss difference derived from the raw and perturbed embeddings (Figure 1- 2⃝); 3) We demonstrate the compatibility of CE-VFAL with communication compression techniques, which can reduce the size of forwarded-transmitted messages (Figure 13⃝). CE-VFAL is the first theoretically verified VFL framework that applies AT. Inspired by the optimal control theorem [Li et al., 2018; Li and Hao, 2018], we constrain the multisource approximate gradients arising from frozen slack variables, gradient estimation, and communication compression. Subsequently, we demonstrate the sublinear convergence rate of model updates through the training process.

Contributions. Our contributions are as follows:

1. We propose the communication-efficient CE-VFAL framework fundamentally reduces client-server interactions through lazy propagation, and integrates advanced compression and ZOO to minimize message sizes, achieving substantial communication efficiency.
2. We provide the first theoretical convergence analysis for VFAL, rigorously characterizing the interaction between multi-source approximation gradients and establishe an O(1/√K) convergence rate over K iterations, matching standard VFL guarantees.
3. Experiments across datasets and architectures verify that CE-VFAL achieves state-of-the-art robustness with substantially lower communication cost than prior VFL adversarial training methods.

2 Preliminaries

We consider a classifier F : X → Y parameterized by weights Θ and predicting a class ˆ y := F(x) := argmaxy∈Y fy(x) for every input x ∈ X with label y ∈ Y in D, where f is a neural network. Notations are summarized in Appendix 2.1. We sort the indexes of sample, client, and network layer, for example, xi,c,t represents i-th sample, c-th client and t-th layer. Sample's index has higher priority than client's, and client's has higher priority than layer's.

Vertical Federated Learning. CE-VFAL applies to various VFL architectures. This paper focuses on the splitV FLc scenario, which typically decomposes neural network f into client models {f1, f2, . . . , fC} parameterized by θ = {θ1, θ2, . . . , θC}, which operate only on local data, and a sever model fs parameterized by ψ. The server party holds Y . Let Θ = (θ, ψ), the global loss can be as:

where ℓ is the loss function, C is the number of client parties, and n is the sample size.

Adversarial Training. AT optimizes the parameters Θ of a neural network f on adversarially perturbed inputs, thereby increasing robustness against test-time attacks [Madry et al., 2017]. This process involves solving a minimax problem on the cross-entropy loss:

where η is the adversarial perturbation, and its magnitude is constrained by ∥η∥ ≤ ϵ, where ϵ is a non-negative scalar that bounds the size of the perturbation. For the inner maximization objective in Equation (2), PGD takes the following step (with step size α) in each iteration:ηt+1 = Π∥η∥≤ϵ(ηt + α g(ηt) ∥g(ηt)∥ ), where g(ηt) = ∇ηL(x + η, y;Θ) is the gradient respect to η, Π∥η∥≤ϵ is a projection to the ϵ-ball.

Definition 1. Vertical Federated Adversarial Learning. Inspired by the idea of viewing AT as a dynamic system [Weinan, 2017], we can describe a T-layer deep neural network by the recursion xt+1 = ft(xt, θt), for t = 0, . . . , T-1, where xt ∈ Rnt represents the states (i.e., the output of the t-th layer), ft : Rnt ×Rmt → Rnt+1 is the state transition map, θt ∈ Rmt are the trainable control parameters, and the initial conditions are provided by the inputs to the network, xi,0. Rewrite (2) as the subsequent optimal control problem:

where Rt is a potential regularizer for the t-th layer.

3 Methodology

In this section, we present CE-VFAL that substantially reduces both communication rounds and message sizes while maintaining model robustness. CE-VFAL introduces three key innovations through a carefully designed hybrid optimization strategy: (1) integration of ZOO to reduce backward communication load, (2) implementation of uniform scale compressor to decrease forward message size, and (3) adaptation of lazy propagation to minimize communication rounds during adversarial sample generation. For the convenience of subsequent analysis, we rewrite loss function as Ai(η, ψ, θ).

Assumptions. The formal definition and detailed discussion of the assumptions are in Appendix 2.2. We make several crucial assumptions: the functions ft, fc, L, and Rt are K-Lipschitz continuous in x, uniformly with respect to θ and ψ, the gradient of the adversarial loss function, ∇ηAi(η, ψ, θ),∇ψAi(θ, ψ, θ) and ∇ηAi(η, ψ, θ) satisfy Lipschitz conditions (Assumption 1); the adversarial loss function Ai(η, ψ, θ) possesses an unbiased gradient (Assumption 2) and is characterized by bounded Hessian matrices Hψ and Hθ (Assumption 3), as well as bounded block-coordinate gradients Qψ and Qθ (Assumption 4); Ai(η, ψ, θ) exhibits zstrong concavity with respect to η (Assumption 5).

3.1 ZOO for Backpropagation

To reducing communication rounds, CE-VFAL reduces the backward message size by employing ZOO (Details in Appendix B.1) within a cascading hybrid optimization strategy. ZOO is applied only during the communication phase, while the rest of the model uses First-Order (FO) gradients. We estimate gradients by sampling random perturbations and evaluating their impact with average random gradient estimation (Avg-RandGradEst).

where tc is the layer number of c-th client, xtc represents the output embeddings of c-th client and µ is the smoothing parameter. The distribution P of u follows the multivariate normal distribution N(0, I), and q is the number of sampling times. Each δj is the loss difference derived from the original and perturbed embeddings. Both server and clients synchronize their random sequences by sharing a common random seed at the start of training. According to (4), the server only needs to communicate a set δ = {δ1 , δ2 , . . . , δq } back to the clients for the gradient estimization. In contrast, FO gradients require transmitting partial derivatives for the entire embedding. The size ∥δ∥ = q is significantly smaller than the size of the partial derivatives, equal to ∥xtc ∥ (that is ∥xtc ∥ ∗B for one batch).

Estimated gradient. ZOO is used to estimate gradients rather than exact calculations, leading to an "Estimated Gradient." The estimation error due to the finite difference approximation is bounded with Lemma 1.

Lemma 1. Bound Zeroth-Order Error. For arbitrary f ∈ C 1L(Rd ), we have:

1) fµ(x) is continuously differentiable, its gradient is LµLipschitz continuous with Lµ ≤ L:

where u is drawn from the uniform distribution over the unit Euclidean sphere, fµ(x) = E(f(x + µu)) is the smooth approximation of f.

2) For any x ∈ Rd, the following inequalities satisfy:

Proof of this lemma is provided in [Liu et al., 2018; Gao et al., 2018].

3.2 Compression for Forward Propagation

To further alleviate the communication load, we showcase the compatibility of CE-VFAL with communication compression techniques (Details in Appendix 1.6). Utilizing the ZOO method, the size of the backward message is reduced to just q. Thus, CE-VFAL substantially reduces the size of the forwardtransmitted messages by employing a uniform scale compressor [Bennett, 1948] C(·)8. In each communication round, the clients transmit compressed embeddings, reducing the byte size from 32 bits to 8 bits, to the server instead of the raw messages.

Definition 2. Compression Error (forward message). Considering sample i, we can define the compression error of Cf : ϵi,c, c = 1, 2, ..., C, i.e. ϵi,c = Cθ(i, xtc ) - xi,tc . We denote the expected norm of the error from the client c at global iteration k as E k i,c = E||ϵki,c ||2 , and E k = maxc E k i,c . Since all client operations are synchronized, the error from all clients is ϵki = (ϵki,1 , ϵki,2 , ..., ϵki,C ). Then, the expected norm of the error from all clients:

Compression gradient. In theM-loop, a compression gradient occurs when the output embeddings are compressed for transmission, resulting in compression error (Assumption 2

and 3, Definition 1). By ensuring the compression preserves the essential gradient information, we show that these errors are bounded and do not impede convergence in Lemma 1.

Lemma 2. Bound Compression Error. Under Assumption 3, 4, and Definition 2, the norm of the difference between the loss function value with and without compression error is bounded:

3.3 Lazy Propagation

VFAL enhances robustness but incurs significant communication costs due to the numerous rounds required for adversarial sample generation. CE-VFAL employs the lazy propagation principle, confining most forward and backward propagation to the first network layer of client models during adversarial updates. We ingeniously leverage lazy propagation, originally designed to reduce computational costs, to minimize the communication rounds, keeping most adversary updates localized. Considering the VFL model as a dynamic system according to (3), the key mathematical insight lies in the computations can be constrained solely to the first layer while updating adversaries, implemented using a slack variable p (i.e. the delayed gradient at the first layer):

where g˜Θ represents the server model and the client models excluding the first layer, and f0 = {f1,0, f2,0, . . . , fC,0} represents the first layer of all C client models with output xi,1 = {xi,1,1, xi,2,1, . . . , xi,C,1}.

For the objective function (3), the inner maximization loop for adversary updating involves an M-loop, with each iteration of the M-loop containing an N-loop. In each Mloop, the slack variable p is calculated by querying the entire model. During the N-loop, the adversarial perturbation η is updated using p and a gradient ascent step, while keeping the network parameters Θ fixed. After M iterations, a single update to Θ is performed in the outer minimization loop using a gradient descent step.

Delayed gradient. This frozen slack variant introduces an oracle error in adversary updating, resulting in a delayed gradient. Given the strong concavity of η in the inner loop, we define the gradient of the loss function concerning η with Assumption 5 and bound it with Lemma 2.

Lemma 2. Bound the gradient for η. Under Assumption 1, Lemma 1, 3, at the i-th sample and k-th iteration, let xi,tc be the output of c-th client and G′ = Ktc+1 (Ktc + tc(tc - 1)K2tc-2 + tcK2tc ), αx < 1 Lηη , we define G = KG′ , αx <

1 Lηη , the pseudo-partial derivative for η satisfies the following

When calculating the gradient of η, we also introduce ZOO error and compression error. Therefore, we divide the gradient of η into two parts. The first part is the delayed gradient calculated within the client. The second part is the gradient which includes ZOO error and compression error, which is generated during the transfer from the server to the client. Finally, we obtain the gradient of η using the chain rule.

3.4 Algorithm

The training scheme of CE-VFAL is outlined in Algorithm 1 and detailed further in Appendix 3. Each client computes the embeddings of its local features, denoted as xm,n i,c,0 , where i denotes the selected sample. These embeddings are transmitted to the server after compressing with C(·)8. The server receives embeddings from all clients and introduces random disturbances to calculate perturbed embeddings and the loss difference {δj i,c }qj=1 . Clients estimate gradients with δ and then calculate the slack variable pmi,c . With pmi,c , client parties updates the adversarial sample N times (Line 10 in Algorithm 1) using gradient ascent. After M times such updating, the server party and client parties update the parameters, denoted as ψ and θ respectively, using gradient descent. This process iterates until convergence criteria are met.

3.5 Communication Efficiency of CE-VFAL

In analyzing the communication efficiency, we focus on communication rounds and load which are shown in Table 1.

• Backward Communication Load: CE-VFAL leverages the ZOO to drastically reduce the communication load during the backward pass. Traditional FO-based VFAL methods require transmitting |xtc | ∗ B elements from the server to the client for each backward propagation. In contrast, CE-VFAL reduces this load to q elements, where q represents the sampling times for multiple point estimations of ZOO method and is typically much smaller than |xtc | ∗ B.
• Forward Communication Load: During the forward pass, CE-VFAL employs a compression technique denoted as C(|xtc | ∗ B)8, which reduces the byte size of forward-transmitted messages from 32 bits to 8 bits.
• Number of Communication Iterations: By integrating N communication-free iterations within each M communication iteration, CE-VFAL significantly reduces the total number of communication iterations required.

Algorithm 1: CE-VFAL-M-N

Input: Learning rates ax, aψ, aθ; Perturbation µ; Random vector u from distribution P; Output: Model parameters Θ = {θ1, θ2, . . . , θC, ψ}; 1 Initialization: Initialize parameters θ1, θ2, . . . , θC, ψ; 2 while not convergent do 3 Randomly select a sample xi; 4 for m = 1 to M do 5 for each client c do 6 Compute embeddings xi,tc , and send C(xi,tc )8 to server; 7 while receiving loss differences {δj i,c }qj=1 do 8 Compute ZO gradient ˆ∇xi,tc lm i (Eq. 4); 9 Compute slack variable pmi (Eq. 5); 10 for n = 1 to N do 11 Update adversarial sample xm,n i,c,0 ; 12 Server receives xi,tc and compute perturbations; 13 Server computes and sends loss differences {δj i }qj=1 ; 14 for each client c do 15 Update local model parameters θc; 16 Update server model parameters ψ;

4 Convergence Analysis

The convergence analysis of CE-VFAL presents unique theoretical challenges arising from the intricate interactions among multiple sources of gradient approximations: (1) delayed gradients induced by lazy propagation mechanism, (2) compression gradients stemming from embedding quantization, and (3) estimated gradients resulting from zeroth-order optimization. These approximations introduces two analytical challenges: precisely characterizing the complex interplay among these various gradient approximations, and developing rigorous bounds that account for their cumulative effects throughout the training process.

gradient of CE-VFAL-M-N satisfies the following:

First-order optimization convergence term:

Stochastic gradient descent term:

Zeroth-order error term:

Compression error term:

Lazy propagation error term:

The term T1 is typical for convergence of first-order optimization algorithms on smooth non-convex functions; The term T2 is typical for stochastic gradient descent at fixed step size; Terms Ec are the errors during forward communication due to compression; Term Ez is the errors due to ZOO; Term Ep is errors due to lazy propagation.

Corollary 1. If we choose αθ and αψ as O( 1 √K ), µ = O( 1 K 14 ), E = O( 1 √K ), Γ = O( 1 √K ), we can derive the sublinear convergence rate:

By constraining multi-source approximate gradients, we demonstrate the sublinear convergence rate O( 1 √K ).

Remark 1. The term O( NM ) reveals the optimal parameter selection strategy: M should be set as large as the communication budget allows since π(M,N) decreases with respect to M. Due to the convexity of π(M,N) in N, the optimal N exists at the point where the partial derivative with respect to N becomes positive-beyond this point, excessive N values deteriorate robust accuracy. This dependence is empirically validated through ablation experiments (Figure 3).

Proof Sketch. We begin by transforming the original minmax optimization problem into a Hamiltonian system (Appendix A.4). The convergence analysis leverages three types of approximate gradients: delayed gradient (Lemma 2), compression gradient (Lemma 3), and estimated gradient (Lemma 1). We establish the global convergence by proving that the loss function is L-smooth (Assumption 1). By combining the results fromM, N, and outer loop analyses, we demonstrate that the model parameters converge asymptotically (Theorem 1). We provided detailed proof in Appendix 2.

5 Experiments

We conducted a systematic suite of experiments to assess the efficacy of our proposed framework. Due to space limitations, experimental details are detailed in Appendix 4. The source code for this project is available at the following URL: https://workelaina.github.io/CE-VFAL/.

5.1 Experiment Setups

Datasets. We evaluate on public datasets including image datasets MNIST [LeCun et al., 1998], FER [Barsoum et al., 2016], CIFAR10 and CIFAR100 [Krizhevsky, 2009], and tabular dataset Criteo [Tien et al., 2014]. To simulate the VFL scenario, we allocated distinct features to each party based on the described methodology in prior works [Luo et al., 2021; Qiu et al., 2022; Fu et al., 2022].

Training procedures. The number of parties is set to three, comprising one server and two clients. The server's model consists of two linear layers, while the client models use ResNet18 for MNIST and FER, ResNet34 for CIFAR10, and MLP for Criteo. Throughout the communication process between the server and clients, the size of the message (i.e., the embedding) for each sample is set to 128.

Baselines. We implemented various AT algorithms within a standard VFL context as baselines, including VFL PGDR [Madry et al., 2017], VFL FreeAT-ϵ [Shafahi et al., 2019], VFL FreeLB-m [Zhu et al., 2019], and Clean VFL [Vepakomma et al., 2018], which represents a standard VFL model without AT. To ensure a fair comparison with CEVFAL-M-N, we constrained the number of communications conducted in a single adversarial sample generation not to exceed PGD's setting, denoted as max{M, R, ϵ, m} = R.

Adversarial attack. To substantiate robustness, we employed a suite of adversarial attack methods: FGSM is a fast and non-iterative attack [Kurakin et al., 2016]; PGD R operates by iteratively perturbing input data using gradients to maximize the model's loss [Madry et al., 2017]; CW incorporates a custom loss function to ensure minimal perturbations while maintaining misclassification [Carlini and Wagner, 2017]; AutoAttack (AA) combines various attack strategies, enhancing the likelihood of evading detection and bypassing defenses [Croce and Hein, 2020].

5.2 Main Results on Various Datasets

Our proposed CE-VFAL consistently delivers strong robustness with substantially lower communication cost across datasets. On MNIST (Table 2), it achieves leading clean/robust accuracy with only ∼1/65 of VFL PGD's communication. On FER (Table 3), CE-VFAL maintains robust performance under multiple attacks while incurring minimal overhead. On CIFAR10 (Table 4), it offers the best overall trade-off, reducing communication to 5.7% of VFL PGD. On Criteo (Table 5), CE-VFAL-5-10 matches clean accuracy while significantly improving robustness, whereas VFL PGD-40 requires ∼65× higher communication for comparable robustness.

Evaluation on Training Time. Figure 2 demonstrates that CE-VFAL-5-3 achieves 80% accuracy within 5,000 seconds, converging 6-8 times faster than baselines, while maintaining a superior balance between efficiency and robustness.

5.3 Ablation Study

The dependence on M and N. We conducted extensive experiments on the MNIST dataset to explore the dependence on parameters M and N. Figure 3 illustrate the change in accuracy with a fixed M = 3, 5, 8, 10, respectively, while varying N. It is evident that performance rapidly degrades with increasing N beyond a certain threshold, as predicted by Corollary 1. This observation highlights the sensitivity of the model's performance to N, emphasizing the importance of optimizing N to maintain high accuracy.

Effect of ZOOParameters. Table 6 investigates how sampling times (q) in ZOO affect model robustness and communication overhead. While ZOO consistently achieves lower communication costs than first-order methods, increasing q substantially improves robustness across different attacks. Although ZOO with maximum sampling falls slightly short of first-order performance on some attacks, it achieves comparable clean accuracy and even superior performance on certain attacks with lower communication overhead.

Effect of Compression Parameters. We investigate the impact of compression by evaluating different compression rates on forward messages. Table 7 reveals an interesting trade-off between compression rate, model robustness, and communication efficiency. Moderate compression achieves comparable clean accuracy to the uncompressed baseline while demonstrating superior robustness and significantly reducing communication costs. However, extreme compression severely degrades model performance across all metrics.

Experiments on different clients. To evaluate CE-VFAL's scalability across different federation scales, we conducted experiments on the Criteo dataset with varying client numbers. Table 8 demonstrates that CE-VFAL maintains superior performance and low communication overhead across different client configurations, exhibiting consistent robustness against various attacks while scaling effectively.

Experiments on different perturbation budgets (ϵ). To evaluate robustness against varying attack intensities, we conducted ablation experiments with different perturbation budgets (ϵ). Table 9 shows that as perturbation budgets increase, all models experience declining defensive capabilities. CEVFAL shows remarkable stability across all perturbation levels, maintaining consistently robustness even under strong attacks while others show substantial degradation.

6 Conclusions

This paper presents CE-VFAL, a novel framework that significantly enhances the robustness of VFL models while maintaining high communication efficiency. By integrating the lazy propagation principle, ZOO, and compression techniques, CE-VFAL achieves a substantial reduction in the frequency and load of client-server interactions. Our theoretical analysis provides the first convergence proof for a VFAL framework, highlighting the sublinear convergence rate and robustness against adversarial attacks. Extensive experiments demonstrate CE-VFAL's superior performance in balancing communication costs with SOTA robustness.

Acknowledgements

Dr. Yi Chang was supported by the National Key R&D Program of China under Grant No. 2023YFF0905400, and the National Natural Science Foundation of China through grants No. U2341229 and No. 62076138. Dr. Zhaogeng Liu was supported by the Young Scientists Fund (C Class) of the National Natural Science Foundation of China under Grant No. 62506142.

Contribution Statement

Tianxing Man and Jinjie Fang (denoted by ∗ on the first page) contributed equally to the conceptualization, methodology, investigation, and writing of this work. Ganyu Wang supported the methodology and review. Yu Bai and Zhaogeng Liu contributed to the evaluation and visualization. Bin Gu and Yi Chang (the corresponding authors, denoted by † on the first page) contributed to the methodology, as well as the review and editing of the manuscript.