Shakerato

Limited GPU Memory GPU usually has lesser device memory than host memoryThe latest high-end GPU (such as NVIDIA GPU P100)12–16 GB device memoryHost system memory256GBTrend for deep learning models is to have a “deeper and wider” architectureEspecially, RNN needs a lot of memory 1. First Solution: distributed Deep LearningSource: M. Cho et al., "PowerAI DDL", 2017PowerAI DDL provides a unified in..