Federated averaging: local training, client weighting and heterogeneous data

What federated averaging means

Federated learning coordinates model training while the training records remain distributed across clients. In federated averaging, or FedAvg, clients train local copies of a shared model and a server aggregates their resulting parameters or updates. The original method was designed around limited communication and uneven, differently distributed client data. [1]

A basic training round

The server distributes the current model to participating clients. Each starts from that same model and performs a specified amount of local optimization. The server then forms an aggregate of the returned models, commonly weighted by the number of training examples contributing at each client. The next round begins from the aggregate. Model architecture, parameter ordering and feature interpretation must match; this is not arbitrary averaging of independently designed networks. [1]

Designing an evaluation

A practical evaluation records the client-sampling policy, local epochs, batch size, learning rate, aggregation weights and failed or delayed contributions. Compare communication rounds, transferred bytes and model quality together. A configuration requiring fewer rounds can still be slower if its local work or communication payload grows.

Preserve realistic client partitions during evaluation. Report both example-weighted performance and a client-level view, with sample counts. This proposed reporting design makes it possible to see whether a global improvement mainly serves the largest clients.

Illustrative weighting example

Suppose all three clients participate, holding 100, 300 and 600 equally weighted training examples. Their normalized sample weights are 0.1, 0.3 and 0.6. For one parameter, local values of 1, 3 and 5 yield 0.1*1 + 0.3*3 + 0.6*5 = 4.0.

An equal-client average would be 3.0, representing a different weighting choice. Neither number establishes predictive quality. This arithmetic example only illustrates aggregation; it is not a training result or an argument that larger clients should always dominate the objective.

Limits and privacy

Clients can differ in class frequencies, feature distributions, label relationships and availability. Such heterogeneity can make local updates pull in different directions. Increasing local training changes the communication–optimization tradeoff and is not a universal improvement. [2]

Local raw-data storage does not make model updates harmless. Secure aggregation can hide individual contributions from the server under its protocol assumptions, but the aggregate and released model can still reveal information. Differential privacy requires an additional, specified mechanism and privacy accounting; it is not an automatic property of FedAvg. [2] No federated training experiment, privacy proof or deployment benchmark is claimed here.

Sources

[1] Communication-Efficient Learning of Deep Networks from Decentralized Data

[2] Advances and Open Problems in Federated Learning