---
title: "Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning"
authors:
  - "Jun Luo"
  - "Shandong Wu"
doi: "10.24963/ijcai.2022/301"
url: "https://ijcai.org/proceedings/2022/301"
pdf: "https://ijcai.org/proceedings/2022/0301.pdf"
venue: "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence"
year: 2022
pages: "2166-2173"
track: "Main Track"
keywords:
  - "Knowledge Representation and Reasoning: Reasong about actions"
  - "Computer Vision: Biomedical Image Analysis"
  - "Data Mining: Federated Learning"
---

# Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning

Jun Luo, Shandong Wu

*Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence*, pages 2166-2173. DOI: [10.24963/ijcai.2022/301](https://doi.org/10.24963/ijcai.2022/301)

## Abstract

Conventional federated learning (FL) trains one global model for a federation of clients with decentralized data, reducing the privacy risk of centralized training. However, the distribution shift across non-IID datasets, often poses a challenge to this one-model-fits-all solution. Personalized FL aims to mitigate this issue systematically. In this work, we propose APPLE, a personalized cross-silo FL framework that adaptively learns how much each client can benefit from other clients’ models. We also introduce a method to flexibly control the focus of training APPLE between global and local objectives. We empirically evaluate our method's convergence and generalization behaviors, and perform extensive experiments on two benchmark datasets and two medical imaging datasets under two non-IID settings. The results show that the proposed personalized FL framework, APPLE, achieves state-of-the-art performance compared to several other personalized FL approaches in the literature. The code is publicly available at https://github.com/ljaiverson/pFL-APPLE.

## Full text

## 1 Introduction

In recent years, federated learning (FL) [McMahan et al., 2017; Kairouz et al., 2019] has shown great potential in training a shared global model for decentralized data. In contrast with previous large-scale machine learning approaches, training in FL resides on the sites of data owners, without the need to migrate the data, which reduces systemic privacy risks and expenses on massive datacenters [Kairouz et al., 2019]. Compared to separate individual training, the leading FL algorithm, FedAvg [McMahan et al., 2017], as a representative of global FL algorithms, attempts to train a consensus global model by iteratively averaging the local updates of the global model. However, such an approach often suffers from convergence challenges brought on by the statistical data heterogeneity [Smith et al., 2017; Li et al., 2020; Hsieh et al., 2020], where data are not identically distributed (non-IID) across all clients due to the inherent diversity [Sahu et al., 2018; Li et al., 2019].

∗An extended version of this paper (with the Appendix included) can be found at https://arxiv.org/abs/2110.08394.

Figure 1: The workflow of the proposed framework, APPLE. Each client trains a local personalized model, uploads its updated core model to the server, and downloads others' core models from the server as needed at the beginning of each round.

![Image](https://ijcai.org/proceedings/2022/fulltext/301/image_000000_569a034827dcf498d5ea5d452a96712cbfe52ad2eb351c3006997bd161d2b7bf.png)

Data heterogeneity lies almost everywhere in real-world FL applications. For instance, in a cross-device training of a mobile keyboard next-word prediction model, the data heterogeneity can be generated by different typing preferences of users [Hard et al., 2018; Yang et al., 2018]; in addition, medical datasets across different silos are heterogeneous by nature, due to factors such as different data acquisition protocols and various local demographics [Rieke et al., 2020]. Moreover, in cross-silo FL, data heterogeneity may lead to inferior performance of federated models in certain silos, which may cost their incentives of further participation in the federation.

Based on the number of trained model(s), attempts to supplement FL algorithms with the ability to better handle data heterogeneity fall into two general schemes. The first scheme tries to enhance the global consensus model for higher robustness to non-IID datasets [Li et al., 2020; Karimireddy et al., 2020; Acar et al., 2020; Zhao et al., 2018]. The other scheme looks at FL from a client-centric perspective, aiming to train multiple models for different clients [Kulkarni et al., 2020; Kairouz et al., 2019], and is often referred to as personalized FL.

Personalized FL tries to systematically mitigate the influence of data heterogeneity, since a different model could be trained for a different target data distribution. Efforts in this direction include approaches that fine-tune the global model [Wang et al., 2019a], and more sophisticated approaches that leverage meta-learning or multi-task learning [Smith et al., 2017; Sattler et al., 2020] to learn the relationships between source and target domains/tasks, which corresponds to different distributions of the datasets. Other efforts pay more attention to interpolations between local models and the global model [Deng et al., 2020; Huang et al., 2021; Zhang et al., 2020].

In this work, we focus on the personalization aspect of cross-silo FL for non-IID data. We propose Adaptive Personalized Cross-Silo Federated Learning (APPLE), a novel personalized FL framework for cross-silo settings that adaptively learns to personalize each client's model by learning how much the client can benefit from other clients' models, according to the local objective. In this process, the clients do not need to acquire information regarding other clients' data distributions. We illustrate the workflow of APPLE in Figure 1.

There are three major distinctions between APPLE and other existing personalized FL algorithms: in APPLE, (1) after local training, a client does not upload the local personalized model, but a constructing component of the personalized model, here called a core model; (2) the central server only maintains the core models uploaded from the clients, for further downloading purposes; (3) a unique set of local weights on each client, here called a directed relationship (DR) vector, is adaptively learned to weight the downloaded core models from the central server. This enables the personalized models to take more advantage of the beneficial core models, while suppressing the less beneficial or potentially harmful core models. We also introduce a method to flexibly control the focus of training APPLE between global and local objectives, by dynamically penalizing the DR vectors. The DR vectors are never uploaded to the server, nor shared with other clients. Coupled with uploading only the core models rather than the personalized models, the privacy risks are further reduced.

We summarize our contribution as follows:

- We propose APPLE, a novel personalized cross-silo FL framework that adaptively learns to personalize the client models.
- Within APPLE, we introduce a method to flexibly control the focus of training between global and local objectives via a dynamic penalty.
- We evaluate APPLE on two benchmark datasets and two medical imaging datasets under two types of non-IID settings. Our results show that APPLE achieves state-ofthe-art performance in both settings compared to other personalized FL approaches in the literature.

## 2 RelatedWork

## 2.1 Federated Learning on Non-IIDData

Federated learning (FL) [McMahan et al., 2017; Kairouz et al., 2019] enables participating clients to collaboratively train a model without migrating the clients' data, which mitigates the systemic privacy risks. The most notable FL algorithm, FedAvg [McMahan et al., 2017], achieves this by aggregating the updated copies of the global model using an averaging approach. While concerns for the behavior of FedAvg on non-IID data began to accumulate [Sahu et al., 2018], numerous works have focused on the robustness of FL on nonIID data. Li et al. [2020] proposed FedProx that penalizes the local update when it is far from the prox-center. Karimireddy et al. [2020] proposed SCAFFOLD that corrects the local gradient under client-drift with control variates. FedDyn by Acar et al. [2020] dynamically updates the regularizer in the empirical risk to reduce the impact of data heterogeneity.

## 2.2 Personalized Federated Learning

To systematically mitigate the impact of data heterogeneity, a new branch of FL, the personalized FL, has emerged in recent years [Kulkarni et al., 2020]. Instead of being restricted by the global consensus model, personalized FL allows different models for different clients, especially when the data are drawn from distinct distributions.

In this line of work, a natural way is to fine-tune the global model at each client [Wang et al., 2019a]. However, Jiang et al. [2019] claimed that fine-tuning the global model may result in poor generalization to unseen data. They also demonstrated the similarity between personalizing a FedAvg-trained model and meta-learning [Finn et al., 2017]. Some recent works have focused on the overlap between meta-learning and FL [Fallah et al., 2020; Khodak et al., 2019].

Apart from approaches that need further fine-tuning the trained models, Smith et al. [2017] proposed MOCHA that leverages multi-task learning to learn the relationship between different clients. Sattler et al. [2020] focused on a new setting where clients are adaptively partitioned into clusters, and a unique personalized model is trained for each cluster of clients.

Other personalized FL algorithms include carefully interpolating a model for each client. APFL [Deng et al., 2020] weights the global and the local model at each client. FedFomo [Zhang et al., 2020] computes estimates of the optimal weights for each client's personalized model using a local validation set. FedAMP [Huang et al., 2021] uses an attention-inducing function to compute an interpolated model as the prox-center for the personalized model.

## 3 Adaptive Personalized Cross-Silo Federated Learning

In this section, we look at personalized FL with more details, and present APPLE, a framework for personalized cross-silo FL that adaptively learns to personalize the client models. Similar to most FL methods, in APPLE, the training progresses in rounds. Each client iteratively downloads and uploads model parameters in each round. However, in APPLE, each client uploads a constructing component of the personalized model, here called a core model. And the central server maintains the core models uploaded from the clients. Before we go into further details, we formulate the problem and define the notations which will be used throughout the paper.

## 3.1 ProblemFormulation

In general, federated learning aims to improve model performance of individually trained models, by collaboratively training a model over a number of participating clients, without migrating the data due to privacy concerns. Specifically, the goal is to minimize:

<!-- formula-not-decoded -->

where fG(·) denotes the global objective. It is computed as the weighted sum of the N local objectives, with N being the number of clients and pi ≥ 0 being the weights. The local objective Fi(·) of client i is often defined as the expected error over all data under local distribution Di, i.e.

<!-- formula-not-decoded -->

where ξ represents the data under local distribution Di. As shown in Equation 2, Fi(·) is often approximated by the local empirical risk on client i using its training set Dtr i (ni = |Dtr i |). The notable FedAvg tries to solve the empirical risk minimization (EMR) by iteratively averaging the local updates of the copy of global model, i.e. wt+1 = ∑Ki=1 piwi t+1 , where K is the number of selected clients in each round, and the pi's are defined as the ratio between the number of data samples on client i, ni, and the number of total data samples from all clients n, with constraints: ∀i ∈ [N], pi ≥ 0, and ∑i pi = 1.

In personalized FL, the global objective slightly changes to a more flexible form:

<!-- formula-not-decoded -->

where fP (·) is the global objective for the personalized algorithms, and W = [w1,w2, ...,wN] is the matrix with all personalized models. In this work, we aim to obtain the optimal W ∗ = argminW fP (W), which equivalently represents the optimal set of personalized models w∗ i , i ∈ [N]. In addition, we focus on the cross-silo setting of FL, which is differentiated with the cross-device setting by much smaller number of participating (stateful) clients, and no selection of clients are strictly needed at the beginning of each round.

## 3.2 Adaptively Learning to Personalize

As mentioned above, in APPLE, each client uploads to the central server a core model, and downloads other clients' core models maintained on the server at the end and beginning of each round, respectively. In an ideal scenario where communication cost is not taken into account, each core model maintained on the server is downloaded by every client. In practice, communication is costly, and the limitations on the communication bandwidth always exist. We will discuss how APPLE handles this in Section 3.4.

In APPLE, after each client has downloaded the needed core models from the server, the personalized model for client i is subsequently computed as

<!-- formula-not-decoded -->

where w(p) i represents the personalized model of client i, and w(c) j is the downloaded client j's core model. Similar to some personalized FL algorithms that focus on interpolating a model for each client [Acar et al., 2020; Zhang et al., 2020], the personalized model here is also a convex combination of models. The difference is that in APPLE, there is a unique set of learnable weights for each client. We use pi,j to denote the learnable weight on client i for the downloaded core model w(c) j , and use pi = [pi,1, ..., pi,N]T to denote the set of learnable weights on client i, calling it the directed relationship (DR) vector.

During local training on client i, after the personalized model w(p) i is computed, we freeze the downloaded core models (w(c) j , j ̸= i), and only update its local core model w(c) i using a gradient-based method, such as local Stochastic Gradient Descent (SGD). Meanwhile, we adaptively update the DR vector, pi, according to the local objective, i.e.

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

Note that after a round of local training is finished, each client only uploads the local core model (w(c) i for client i, i ∈ [N]) to the server. The DR vector pi is always maintained at client i without any migration, which makes it impossible for others to infer the personalized model, and further protects the data privacy.

## 3.3 Proximal Directed Relationships

In APPLE, for each client, the learned global information is blended in the downloaded core models, whose contributions to the local personalized model are measured by the learnable weights in the DR vector. Ideally, the entry pi,i, or "selfrelationship", should be larger than the other entries in pi, since the local core model w(c) i is the only network trained with local distribution Di. On the other hand, for all j ̸= i, pi,j should be somewhere in between 0 and pi,i, if the local personalized model w(p) i can benefit more from w(c) j (may happen if the distributions Di and Dj are similar), while pi,j should be closer to 0 or even negative, if w(c) j results in potential negative transfer to w(p) i .

However, in a real-world situation, due to the data heterogeneity in FL, chances for similar distributions among clients are slim. Most off-diagonal entries in the DR matrix should be small. Without any constraint, this may result in a natural pitfall that the learned DR matrix is too quickly drawn to somewhere near the identity matrix (in terms of the Frobenius norm). This can lead the personalized models to hardly benefit from FL, and the training process undesirably resembles individual learning.

## Algorithm 1 APPLE

Input: N clients, learning rates η1, η2, number of total rounds R, proximal term coefficients λ(r), µ, prox-center p0 Output: Personalizedmodelsw(p) 1 ,w(p) 2 , ...,w(p) N on site of corresponding client.

- 1: ∀i ∈ [N], randomly initialize core model w(c) i on server
- 2: ∀i ∈ [N] in parallel, initialize local DR vector pi
- 3: for r ← 1, 2, ...,R do
- 4: for i ← 1, 2, ..., N in parallel do
- 5: Download core models from server as needed.
- 6: Iteratively optimize the local core model w(c) i and local DR vector pi by the following:
- 7: Compute personalized model w(p) i by Eq. (4)
- 8: Compute empirical risk Fi(w(p) i ) by Eq. (7)
- 9: Update w(c) i and DR vector pi by Eqs. (5, 6)
- 10: When optimization is finished, upload local core model w(c) i to the server
- 11: end for
- 12: end for
- 13: return w(p) 1 ,w(p) 2 , ...,w(p) N

To address this issue and facilitate collaboration between clients, we penalize the directed relationship by adding a proximal term [Rockafellar, 1976] to the local empirical risk, which is summarized in APPLE in Equation 7.

<!-- formula-not-decoded -->

In Equation 7, λ is a dynamic function ranging from 0 and 1, with respect to the round number, r, and µ is a scalar coefficient for the proximal term. The prox-center is at p0 = [n1/n, ..., nN/n]. It is obvious that FedAvg is a special case of APPLE by setting µ to ∞, which infers that a larger coefficient of the proximal term can push the personalized model to a global model, facilitating collaboration between clients. While this may benefit the personalized model to learn highlevel features, it is not always desired throughout the training. Ultimately, with the learned high-level features, the personalized models should still focus on how to be personalized.

To this end, we design the function λ(·) with a certain type of decay in terms of the round number, r, inspired by Wang et al. [2019b], and call such λ(r) a loss scheduler. More details regarding the loss scheduler are presented in Appendix A.1. We summarize the steps of APPLE in Algorithm 1.

## 3.4 APPLE under Limited Communication Budget

The collaboration between clients in APPLE relies on the downloaded core models at the beginning of each round. Without considering communication limitations, each client can download all other clients' core models, which maximizes the potential pair-wise collaboration between clients.

However, although the number of clients in cross-silo settings will not be as large compared to the cross-device settings, the communication cost of downloading all core models to each client is still considerable. While this issue can be mitigated by techniques including quantization [Xu et al., 2018] and knowledge distillation [Hinton et al., 2015], in the worst-case scenario, the communication per round still cost N times more overhead than algorithms that only download one model for each client per round (e.g. FedAvg).

To address this issue, we restrict the number of models a client can download per round, denoted by M. Under limited communication budget (1 ≤ M ≤ N - 1), briefly, APPLE decides which M core models to download to each client by the following rules: on client i, the core model of client j will be downloaded if it has never been downloaded on client i (breaks tie randomly); if all other clients' core models have all been downloaded at least once, with high probability, priority goes to client j's core model who has a large pi,j. We elaborate this process in Appendix A.2.

## 4 Experiments

In this section, we demonstrate the effectiveness of APPLE with experiments under two different non-IID federated settings. We show the empirical convergence behaviors and the generalization performances with respect to each client on different image datasets. In addition, we study the transition of the pairwise directed relationships between different clients throughout the training process. Last but not least, we also investigate the performance of APPLE under different levels of limited communication budget.

To evaluate the convergence behavior, we plot the training losses and test accuracies against the number of trained rounds. For the personalized methods, the training loss and test accuracy are computed in a way such that if a data sample resides on client i, then we use the personalized model of client i to conduct inference with it. In addition, we quantify the performance of the methods by computing the test accuracies with respect to each client, and the best mean client test accuracy (BMCTA) (best over all rounds, mean over all clients), a metric also used by Huang et al. [2021].

## 4.1 Experimental Setup

Datasets. We use four public datasets including two benchmark datasets: MNIST and CIFAR10, and two medical imaging datasets from the MedMNIST datasets collection [Yang et al., 2021], namely the OrganMNIST(axial) dataset: an 11-class of liver tumor image dataset, and the PathMNIST dataset: a 9-class colorectal cancer image dataset. We partition each dataset into a training set and a test set with the same distributions (if such split does not pre-exist). Then we transform the datasets according to a non-IID distribution, and ensure the same distributions of local training and local test set.

Pathological and Practical Non-IID Settings. We design two non-IID distributions for empirical evaluation, namely the pathological non-IID and the practical non-IID. For the pathological non-IID, we follow precedent work and select two random classes for each client. A random percentage of images from each of the two selected classes is assigned to the client. To simulate a cross-silo setting, the number of clients is set to be 12. For the practical non-IID, our endeavor aims to simulate a more realistic cross-silo federation of medical institutes. To this end, we partition each class of the dataset into 12 shards (corresponding to the 12 clients): 10 shards of 1%, 1 shard of 10% and 1 shard of 80% images within this class. A randomly picked shard from each class is assigned to each client, so that every client will possess data from every class. The practical non-IID setting is more similar to the real-world FL in medical applications. This is because the datasets at medical institutes most likely contain a variety of categories of data. And due to different demographic distributions of patients, medical institutes located in different regions may have more frequent occurrences of different categories of data. As a result, these datasets are often imbalanced with different majority classes, and vary largely in size. Appendix B.1 shows the data distributions in further details.

Figure 2: The training loss, test accuracy and client test accuracies of personalized methods under the pathological non-IID setting.

![Image](https://ijcai.org/proceedings/2022/fulltext/301/image_000001_f89ebca98838c4ebbeed85e1bc44ea82b0d5f634337e2f7a81f5718abae8135f.png)

Compared Baselines. We compare APPLE, with and without proximal DR penalty, against the following approaches: (1) Separate training, meaning the clients' models are trained purely locally without FL; (2) the FedAvg [McMahan et al., 2017] which takes the average of the locally trained copies of the global model; (3) FedAvg-local, a na¨ıve personalized approach of using the locally trained copy of FedAvg's global model; (4) a fine-tuning approach [Wang et al., 2019a] on the FedAvg and on the FedProx [Li et al., 2020], here denoted as FedAvg-FT and FedProx-FT, respectively; (5) APFL [Deng et al., 2020], a personalized method using a mixture of the global and the local model; (6) HeurFedAMP [Huang et al., 2021], a personalized method on the cross-silo setting with federated attentive message passing; and (7) FedFomo [Zhang et al., 2020], a personalized method that computes first-order approximations for the personalized models. We train each method 160 rounds with 5 local epochs and summarize the results as follows.

## 4.2 Experimental Results

We summarize the empirical convergence behavior and performance under the pathological and the practical non-IID settings in Figure 2, Table 1 and Figure 6 (in Appendix B.2). Across all datasets and non-IID settings, our proposed method has fast convergence behaviors, and achieves highest BMCTAs. Specifically, under the pathological non-IID setting, the separate training reaches comparable performance with other methods, due to little similarity in data distribution shared by different clients, and the small number of classes in each client. With a direct averaging of the local updates as in FedAvg, and no fine-tuning as in FedAvg-FT and FedProx-FT, the global model is hardly able to boost the performance of separate training. The personalized FL methods bring further improvement to the na¨ıve personalization, and APPLE outperforms the other compared personalized FL methods. For the practical non-IID setting, since Practical non-IID MNIST CIFAR10

| Pathological non-IID   |   MNIST |   CIFAR10 |   Organ- MNIST (axial) |   Path- MNIST |
|------------------------|---------|-----------|------------------------|---------------|
| Separate               |   97.34 |     74.96 |                  93.14 |         87.09 |
| FedAvg                 |   95.71 |     51.44 |                  59.43 |         56.61 |
| FedAvg-local           |   99.52 |     90.10 |                  96.76 |         93.21 |
| FedAvg-FT              |   99.43 |     90.49 |                  97.03 |         92.31 |
| FedProx-FT             |   99.43 |     90.49 |                  97.03 |         92.38 |
| APFL                   |   99.75 |     89.30 |                  98.72 |         94.98 |
| HeurFedAMP             |   98.13 |     91.10 |                  98.39 |         96.55 |
| FedFomo                |   99.71 |     91.96 |                  99.31 |         97.24 |
| APPLE, µ = 0           |   99.73 |     92.22 |                  99.66 |         96.78 |
| APPLE, µ ̸= 0          |   99.77 |     92.68 |                  99.61 |         97.51 |

Organ-

MNIST

(axial)

Path-

MNIST

Separate

78.20 63.06 65.21 61.36

FedAvg

94.00 34.32 86.56 53.83

FedAvg-local 97.47 71.99 93.75 78.70

FedAvg-FT

97.66 72.08 94.13 78.69

FedProx-FT 97.66 72.08 94.13 78.69

APFL

98.80 71.19 95.53 86.35

HeurFedAMP 97.45 69.54 86.82 79.33

FedFomo

98.05 70.15 82.86 79.39

APPLE, µ = 0

99.00 75.62 95.70 84.22

APPLE, µ ̸= 0

98.97 77.41 95.62 86.39

Table 1: Best mean client test accuracy (BMCTA) of the four datasets under the pathological and practical non-IID settings. Highest performance is represented in bold.

the local data distributions across all clients are more complicated in terms of the number of classes, and local majority classes, a high performance requires more careful integration of the global information. As a result, the fine-tuning methods outperform some personalized methods, and APPLE also reaches state-of-the-art performance in all settings.

Next, we report the performance of APPLE under limited communication budget. With the same levels of communication restriction, we compare APPLE against FedFomo since FedFomo also needs to download N - 1 models to each client by default. We restrict the maximum number of downloaded models for each client per round (denoted by M) to different levels (M = 11, 7, 5, 2, 1). Table 2 shows the results under these settings. For the pathological non-IID setting, results are mixed across different datasets. APPLE outperforms FedFomo on the CIFAR10 and OrganMNIST (axial) dataset, while FedFomo reaches higher performance on the PathMNIST dataset. For the practical non-IID setting, APPLE outperforms FedFomo across all datasets and different levels of limitations. Note that in APPLE, less downloaded models (smaller M) do not necessarily lead to an inferior performance. This is because the proposed rule of picking which models to download tends to download to a client the top M core models that have the highest chance to benefit the client.

Furthermore, we visualize the change of the directed re- Practical non-IID

| Pathological non-IID   | Pathological non-IID   | MNIST       | CIFAR10     | Organ- MNIST (axial)   | Path- MNIST   |
|------------------------|------------------------|-------------|-------------|------------------------|---------------|
| M =                    | 11 FedFomo APPLE       | 99.71 99.73 | 91.96 92.22 | 99.31 99.66            | 97.24 96.78   |
| M = 7                  | FedFomo APPLE          | 99.71 99.73 | 91.95 92.17 | 99.31 99.53            | 97.33 97.15   |
| M = 5                  | FedFomo APPLE          | 99.71 99.72 | 91.94 92.28 | 99.31 99.48            | 97.40 97.17   |
| M = 2                  | FedFomo APPLE          | 99.71 99.70 | 91.98 92.41 | 99.31 99.47            | 97.25 97.11   |
| M = 1                  | FedFomo APPLE          | 99.71 99.66 | 91.95 92.31 | 99.31 99.59            | 97.15 96.29   |

MNIST CIFAR10

Organ-

MNIST

(axial)

Path-

MNIST

M = 11 FedFomo 98.05 70.15 82.86 79.39

APPLE 99.00 75.62 95.70 84.22

M = 7 FedFomo 97.65 70.24 80.88 80.19

APPLE 98.70 76.14 94.21 84.07

M = 5 FedFomo 97.47 70.44 82.83 79.62

APPLE 98.45 75.63 94.49 85.46

M = 2 FedFomo 96.51 69.87 79.53 79.26

APPLE 98.29 74.84 92.29 84.64

M = 1 FedFomo 91.54 69.93 78.37 75.17

APPLE 98.52 73.03 93.55 83.35

Table 2: BMCTA of APPLE (µ = 0) and FedFomo with a maximum of M models (N = 12) to download for each client per round under the pathological non-IID setting.

lationships throughout the training process. Specifically, we study the local DRs on client 1, i.e. p1,i's, and the pi,i's for all clients. The visualizations are shown in Figure 3 and Figure 8 (in Appendix B.3) for the trajectories of DRs under the pathological and the practical non-IID settings, respectively. As mentioned in Section 3.3, the self-relationship, pi,i, should be larger than the other local DRs, since on client i, the only updated core model is w(c) i . And depending on the similarity between the data distributions on client i and on client j, larger similarity will push pi,j towards |pi,i| and lower similarity will push |pi,j| towards 0. Figure 3 empirically shows these properties. For instance, in CIFAR10, even in the same class, images can have high variance, different client shares little similarity in distribution, so the pi,i's are large and pi,j 's are small. In PathMNIST, p1,6 and p1,8 are closer to p1,1, which makes sense since client 1 and 6 both have a large portion of images from class 1, and client 1 and 8 both have a large portion of images from class 7 (refer to Figure 4 in Appendix B.1 for the data distribution).

Figure 3: Directed Relationships of different datasets under the pathological non-IID setting. The first row shows the DRs on client 1. The second row shows the "self-relationships", pi,i, for each client.

![Image](https://ijcai.org/proceedings/2022/fulltext/301/image_000002_7012ecaecdf40770e4b30b1fd80568c8987017412bc76a87b49f26bcd2b0949e.png)

## 5 Conclusions

In this work, we proposed APPLE, a novel personalized cross-silo federated learning framework for non-IID data, that adaptively learns the quantification of how much each client can benefit from other clients' models, i.e. the directed relationships (DRs). We introduced a proximal DR penalty to control the training between global and local objectives. In addition, we evaluated our method's convergence and generalization behaviors on four image datasets under two nonIID settings. The results showed the overall superior effects of APPLE over several related personalized FL alternatives. Through the visualization of the DRs, our study empirically shows that APPLE enables clients to adaptively take more advantage from other clients with similar distributions, while mitigating the potential non-beneficial influences or negative transfers from clients with drastically different data distributions. We also demonstrated that under limited communication budget, APPLE can still reach state-of-the-art performance. As future work, we plan to further evaluate our algorithm's robustness with real-world medical datasets.

## Acknowledgements

This work was supported in part by a National Institutes of Health (NIH) / National Cancer Institute (NCI) grant (1R01CA218405), a National Science Foundation (NSF) grant (CICI: SIVD: 2115082), the grant 1R01EB032896 as part of the NSF/NIH Smart Health and Biomedical Research in the Era of Artificial Intelligence and Advanced Data Science Program, a Pitt Momentum Funds scaling award (Pittsburgh Center for AI Innovation in Medical Imaging), and an Amazon AWS Machine Learning Research Award. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by NSF grant number ACI-1548562. Specifically, it used the Bridges-2 system, which is supported by NSF award number ACI1928147, at the Pittsburgh Supercomputing Center.

## References

1. [Acar et al., 2020] Durmus Alp Emre Acar, Yue Zhao, Ra- mon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dy- namic regularization. In International Conference on Learning Representations, 2020.
2. [Deng et al., 2020] Yuyang Deng, Mohammad Mahdi Ka- mani, and Mehrdad Mahdavi. Adaptive personalized fed- erated learning. arXiv preprint arXiv:2003.13461, 2020.
3. [Fallah et al., 2020] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learn- ing: A meta-learning approach. arXiv preprint arXiv:2002.07948, 2020.
4. [Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
5. [Hard et al., 2018] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Franc ¸oise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.
6. [Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
7. [Hsieh et al., 2020] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Con- ference on Machine Learning, pages 4387-4398. PMLR, 2020.
8. [Huang et al., 2021] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non- iid data. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 35, pages 7865-7873, 2021.
9. [Jiang et al., 2019] Yihan Jiang, Jakub Konečnỳ, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488, 2019.
10. [Kairouz et al., 2019] Peter Kairouz, H Brendan McMa- han, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.
11. [Karimireddy et al., 2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebas- tian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.
12. [Khodak et al., 2019] Mikhail Khodak, Maria-Florina Bal- can, and Ameet Talwalkar. Adaptive gradient-based meta- learning methods. arXiv preprint arXiv:1906.02717, 2019.
13. [Kulkarni et al., 2020] Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pages 794-797. IEEE, 2020.
14. [Li et al., 2019] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.
15. [Li et al., 2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Pro- ceedings of Machine Learning and Systems, 2:429-450, 2020.
16. [McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
17. [Rieke et al., 2020] Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyri- don Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al. The future of digital health with feder- ated learning. NPJ digital medicine, 3(1):1-7, 2020.
18. [Rockafellar, 1976] R Tyrrell Rockafellar. Monotone opera- tors and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877-898, 1976.
19. [Sahu et al., 2018] Anit Kumar Sahu, Tian Li, Maziar San- jabi, Manzil Zaheer, Ameet Talwalkar, and Virginia Smith. On the convergence of federated optimization in heteroge- neous networks. arXiv preprint arXiv:1812.06127, 3:3, 2018.
20. [Sattler et al., 2020] Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 2020.
21. [Smith et al., 2017] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. arXiv preprint arXiv:1705.10467, 2017.
22. [Wang et al., 2019a] Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Franc ¸oise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252, 2019.
23. [Wang et al., 2019b] Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5017-5026, 2019.
24. [Xu et al., 2018] Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, and Hongkai Xiong. Deep neural net- work compression with single and multiple level quantiza- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
25. [Yang et al., 2018] Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Franc ¸oise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903, 2018.
26. [Yang et al., 2021] Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In 2021 IEEE 18th International Symposium on Biomedical Imag- ing (ISBI), pages 191-195. IEEE, 2021.
27. [Zhang et al., 2020] Michael Zhang, Karan Sapra, Sanja Fi- dler, Serena Yeung, and Jose M Alvarez. Personalized fed- erated learning with first order model optimization. arXiv preprint arXiv:2012.08565, 2020.
28. [Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.