Descending through a Crowded Valley -- Benchmarking Deep Learning Optimizers (Paper Explained)

#ai #research #optimization Deep Learning famously gives rise to very complex, non-linear optimization problems that cannot be solved analytically. Therefore, the choice of a suitable optimization algorithm can often make or break the training of a Deep Neural Network. Yet, the literature is full with hundreds of different algorithms, each claiming to be superior and selecting one of them is mostly done based on popular opinion or anecdotes. This paper investigates 14 of the most popular optimizers in a st