Distributed Deep Learning with ChainerMN

ChainerMN enables multi-node distributed deep learning with the following features:

  • Scalable — it makes full use of the latest technologies such as NVIDIA NCCL and CUDA-Aware MPI,
  • Flexible — even dynamic neural networks can be trained in parallel thanks to Chainer’s flexibility, and
  • Easy — minimal changes to existing user code are required.

This blog post provides our benchmark results using up to 128 GPUs.

ChainerMN can be used for both inner-node (i.e., multiple GPUs inside a node) and inter-node settings. For inter-node settings, we highly recommend to use high-speed interconnects such as InfiniBand.

ChainerMN examples are available on GitHub. These examples are based on the examples of Chainer and the differences are highlighted.