Gradients without Backpropagation

Molto interessante e ottima idea per ridurre il costo di training delle rete neurali.

Bisognerà vedere in che ambiti potra’ funzionare bene.


Using backpropagation to compute gradients ofobjective functions for optimization has remaineda mainstay of machine learning. Backpropagation,or reverse-mode differentiation, is a special casewithin the general family of automatic differen-tiation algorithms that also includes the forwardmode. We present a method to compute gradientsbased solely on the directional derivative that onecan compute exactly and efficiently via the for-ward mode. We call this formulation the forwardgradient, an unbiased estimate of the gradientthat can be evaluated in a single forward run of thefunction, entirely eliminating the need for back-propagation in gradient descent. We demonstrateforward gradient descent in a range of problems,showing substantial savings in computation andenabling training up to twice as fast in some cases.

Continua qui: 2202.08587.pdf

If you like this post, please consider sharing it.

Leave a Comment

Your email address will not be published. Required fields are marked *