Gradients without Backpropagation

Molto interessante e ottima idea per ridurre il costo di training delle rete neurali.

Bisognerà vedere in che ambiti potra’ funzionare bene.

Source: Arxiv.org

Using backpropagation to compute gradients ofobjective functions for optimization has remaineda mainstay of machine learning. Backpropagation,or reverse-mode differentiation, is a special casewithin the general family of automatic differen-tiation algorithms that also includes the forwardmode. We present a method to compute gradientsbased solely on the directional derivative that onecan compute exactly and efficiently via the for-ward mode. We call this formulation the forwardgradient, an unbiased estimate of the gradientthat can be evaluated in a single forward run of thefunction, entirely eliminating the need for back-propagation in gradient descent. We demonstrateforward gradient descent in a range of problems,showing substantial savings in computation andenabling training up to twice as fast in some cases.

Continua qui: 2202.08587.pdf

If you like this post, please consider sharing it.

Leave a Comment

Your email address will not be published. Required fields are marked *