Browsing by Author "Sasse, Robin"

Now showing 1 - 1 of 1

Open Access
Investigating the influence of learning rates on the learning speed of neural networks
(2021) Sasse, Robin
This Bachelor’s Thesis investigates the effects of learning rates on the learning speed of Residual Neural Networks, training on the CIFAR-10 and CIFAR-100 data sets. Besides the optimal constant learning rate setting, we discuss the option of learning rate scheduling and calculating the learning rate. Cyclical schedules with large maximum learning rates are used to recreate a phenomenon called super-convergence, which speeds up the training procedure by as much as orders of magnitude and leads to better generalization capabilities of the network. We present an intuition as to why cyclical learning rates lead to better regularization of the network. We show that super-convergence can be reproduced for the optimizer Adam by introducing cyclical learning rates. Lastly, a method which calculates the learning rate, rather than requiring it as a hyper-parameter, is investigated. This algorithm promises to use statistical element-wise curvature information to automatically tune the learning rate for each iteration and each parameter separately. We show that while the approach of calculating the learning rate is valid, it neither leads to super-convergence nor to a higher validation accuracy achieved by the network when compared to the ones trained with cyclical learning rates.