Regression from linear models to neural networks : double descent, active learning, and sampling

Abstract

Regression, that is, the approximation of functions from (noisy) data, is a ubiquitous task in machine learning and beyond. In this thesis, we study regression in three different settings. First, we study the double descent phenomenon in non-degenerate unregularized linear regression models, proving that these models are always very noise-sensitive when the number of parameters is close to the number of samples. Second, we study batch active learning algorithms for neural network regression from a more applied perspective: We introduce a framework for building existing and new algorithms and provide a large-scale benchmark showing that a new algorithm can achieve state-of-the-art performance. Third, we study convergence rates for non-log-concave sampling and log-partition estimation algorithms, including approximation-based methods, and prove many results on optimal rates, efficiently achievable rates, multi-regime behaviors, reductions, and the relation to optimization.