Proximal boosting and its acceleration

Erwan Fouillen (1) , Claire Boyer (1, 2) , Maxime Sangnier (1, 3)

1 LPSM (UMR_8001) - Laboratoire de Probabilités, Statistique et Modélisation
2 DMA - Département de Mathématiques et Applications - ENS Paris
3 MLIA - Machine Learning and Information Access

Gradient boosting is a prediction method that iteratively combines weak learners to produce a complex and accurate model. From an optimization point of view, the learning procedure of gradient boosting mimics a gradient descent on a functional variable. This paper proposes to build upon the proximal point algorithm when the empirical risk to minimize is not differentiable to introduce a novel boosting approach, called proximal boosting. Besides being motivated by non-differentiable optimization, the proposed algorithm benefits from Nesterov’s acceleration in the same way as gradient boosting [Biau et al., 2018]. This leads to a variant, called accelerated proximal boosting. Advantages of leveraging proximal methods for boosting are illustrated by numerical experiments on simulated and real-world data. In particular, we exhibit a favorable comparison over gradient boosting regarding convergence rate and prediction accuracy.

Domains: Machine Learning [stat.ML]; Machine Learning [cs.LG]

Erwan Fouillen, Claire Boyer, Maxime Sangnier. Proximal boosting and its acceleration. 2020. ⟨hal-01853244v2⟩