Distributed Deep Learning with ChainerMN¶
ChainerMN enables multi-node distributed deep learning with the following features:
Scalable — it makes full use of the latest technologies such as NVIDIA NCCL and CUDA-Aware MPI,
Flexible — even dynamic neural networks can be trained in parallel thanks to Chainer’s flexibility, and
Easy — minimal changes to existing user code are required.
This blog post provides our benchmark results using up to 128 GPUs.
ChainerMN can be used for both inner-node (i.e., multiple GPUs inside a node) and inter-node settings. For inter-node settings, we highly recommend to use high-speed interconnects such as InfiniBand.
ChainerMN examples are available on GitHub. These examples are based on the examples of Chainer and the differences are highlighted.
- Model Parallel
- API Reference