We give an overview of how to utilize multiple computational nodes in the training of a ML model. This question either arises because the model is too large to be handled on a single node, or because the data is inherently local to the clients (for example for privacy reasons) as it is the case for federated learning, which we will focus on. The field is young enough to be fresh and nteresting, but old enough to have established survey papers:

References:

• https://arxiv.org/abs/1912.04977
• https://arxiv.org/abs/1908.07873
• https://arxiv.org/abs/2107.06917
• https://arxiv.org/abs/1706.02677