Kavli Affiliate: Wei Gao
| First 5 Authors: Haoming Wang, Wei Gao, , ,
| Summary:
Federated Learning (FL) can be affected by data and device heterogeneities,
caused by clients’ different local data distributions and latencies in
uploading model updates (i.e., staleness). Traditional schemes consider these
heterogeneities as two separate and independent aspects, but this assumption is
unrealistic in practical FL scenarios where these heterogeneities are
intertwined. In these cases, traditional FL schemes are ineffective, and a
better approach is to convert a stale model update into a unstale one. In this
paper, we present a new FL framework that ensures the accuracy and
computational efficiency of this conversion, hence effectively tackling the
intertwined heterogeneities that may cause unlimited staleness in model
updates. Our basic idea is to estimate the distributions of clients’ local
training data from their uploaded stale model updates, and use these
estimations to compute unstale client model updates. In this way, our approach
does not require any auxiliary dataset nor the clients’ local models to be
fully trained, and does not incur any additional computation or communication
overhead at client devices. We compared our approach with the existing FL
strategies on mainstream datasets and models, and showed that our approach can
improve the trained model accuracy by up to 25% and reduce the number of
required training epochs by up to 35%. Source codes can be found at:
https://github.com/pittisl/FL-with-intertwined-heterogeneity.
| Search Query: ArXiv Query: search_query=au:”Wei Gao”&id_list=&start=0&max_results=3