Model Parallel

• Overview
  • Model Parallelism
  • Philosophy
  • References
• Model Parallel on ChainerMN
  • Step 1: Communicators
  • Step 2: Datasets and Iterators
  • Step 3: Define Communications
  • Note: Define-by-Run and Model Parallelism
  • Note: Delegate Variable and Pseudo Connect
• Example 1: Simple MLP
• Example 2: seq2seq
• Example 3: Channel-wise Parallel Convolution
• Example 4: Ensemble