Scalable and Practical Natural Gradient for Large-Scale Deep Learning

Scalable and Practical Natural Gradient for Large-Scale Deep Learning
Scalable and Practical Natural Gradient for Large-Scale Deep Learning
Other Titles:
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publication Date:
23 June 2020
Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications of batch normalization. We propose Scalable and Practical Natural Gradient Descent (SP-NGD), a principled approach for training models that allows them to attain similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence. Furthermore, SP-NGD scales to large mini-batch sizes with a negligible computational overhead as compared to first-order methods. We evaluated SP-NGD on a benchmark task where highly optimized first-order methods are available as references: training a ResNet-50 model for image classification on ImageNet. We demonstrate convergence to a top-1 validation accuracy of 75.4% in 5.5 minutes using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of 74.9% with an extremely large mini-batch size of 131,072 in 873 steps of SP-NGD.
License type:
Funding Info:
Computational resource of AI Bridging Cloud Infrastructure (ABCI) was awarded by ”ABCI Grand Challenge” Program, National Institute of Advanced Industrial Science and Technology (AIST). This work was supported by JSPS KAKENHI Grant Number JP18H03248 and JP19J13477. This work is supported by JST CREST Grant Number JPMJCR19F5, Japan. (Part of) This work is conducted as research activities of AIST - Tokyo Tech Real World Big-Data Computation Open Innovation Laboratory (RWBC-OIL). This work is supported by ”Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures” in Japan (Project ID: jh180012-NAHI). This research used computational resources of the HPCI system provided by (the names of the HPCI System Providers) through the HPCI System Research Project (Project ID:hp190122)
© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Files uploaded:

File Size Format Action
09123671.pdf 3.07 MB PDF Open