Kavli Affiliate: Jia Liu
| First 5 Authors: Peizhong Ju, Haibo Yang, Jia Liu, Yingbin Liang, Ness Shroff
| Summary:
Federated Learning (FL) has gained significant popularity due to its
effectiveness in training machine learning models across diverse sites without
requiring direct data sharing. While various algorithms along with their
optimization analyses have shown that FL with local updates is a
communication-efficient distributed learning framework, the generalization
performance of FL with local updates has received comparatively less attention.
This lack of investigation can be attributed to the complex interplay between
data heterogeneity and infrequent communication due to the local updates within
the FL framework. This motivates us to investigate a fundamental question in
FL: Can we quantify the impact of data heterogeneity and local updates on the
generalization performance for FL as the learning process evolves? To this end,
we conduct a comprehensive theoretical study of FL’s generalization performance
using a linear model as the first step, where the data heterogeneity is
considered for both the stationary and online/non-stationary cases. By
providing closed-form expressions of the model error, we rigorously quantify
the impact of the number of the local updates (denoted as $K$) under three
settings ($K=1$, $K<infty$, and $K=infty$) and show how the generalization
performance evolves with the number of rounds $t$. Our investigation also
provides a comprehensive understanding of how different configurations
(including the number of model parameters $p$ and the number of training
samples $n$) contribute to the overall generalization performance, thus
shedding new insights (such as benign overfitting) for implementing FL over
networks.
| Search Query: ArXiv Query: search_query=au:”Jia Liu”&id_list=&start=0&max_results=3