The Unusual Effectiveness of Averaging in GAN Training

The Unusual Effectiveness of Averaging in GAN Training
Title:
The Unusual Effectiveness of Averaging in GAN Training
Other Titles:
International Conference on Learning Representations (2019)
DOI:
Publication Date:
01 May 2019
Citation:
Abstract:
We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide the -- to our knowledge -- first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -- mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -- to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.
License type:
PublisherCopyrights
Funding Info:
Description:
The code is available at https://github.com/yasinyazici/EMA_GAN
ISBN:

Files uploaded:
File Size Format Action
There are no attached files.