Whitened gradient descent, a new updating method for optimizers in deep neural networks

Optimizers are vital components of deep neural networks that perform weight updates. This paper introduces a new updating method for optimizers based on gradient descent, called whitened gradient descent (WGD). This method is easy to implement and can be used in every optimizer based on the gradient descent algorithm. It does not increase the training time of the network significantly. This method smooths the training curve and improves classification metrics. To evaluate the proposed algorithm, we performed ۴۸ different tests on two datasets, Cifar۱۰۰ and Animals-۱۰, using three network structures, including densenet۱۲۱, resnet۱۸, and resnet۵۰. The experiments show that using the WGD method in gradient descent based optimizers, improves the classification results significantly. For example, integrating WGD in RAdam optimizer increased the accuracy of DenseNet from ۸۷.۶۹% to ۹۰.۰۲% on the Animals-۱۰ dataset.

کلیدواژه ها:

deep learning ، Optimizer ، Whitened Gradient Descent ، Momentum