Comparison of different Hyperparameter-Tuners for Support Vector Machines : an analysis using Parallel Least-Squares SVM Library on GPU

Abstract

Working with large datasets requires sophisticated tools. One such tool developed for classification is the Support Vector Machine (SVM). As with any ML algorithm, the user has to set several different Hyper Parameter (HP) to run a SVM. Finding the optimal choice of HPs is important for model performance and it is highly dependent on the dataset. Given the number of different HPs, a search space might be massive, so optimization methods have been developed, to automate this search. This work aims to compare three popular choices: The Grid Search, the Random Search and Bayesian Model Search. They are compared in different metrics, such as performance, runtime and energy. Optuna [ASY+19] was used as optimizer backend, it implements all three optimizer types, it implements Tree-Parzan Estimator (TPE) as Bayesian Search algorithm. It was connected to Parallel Least-Squares Support Vector Machine (PLSSVM) [VCBP22] as SVM implementation. PLSSVM can efficiently exploit parallel compute cores. The optimizers have been tested on a selection of different search spaces and datasets with PLSSVM running on Graphic Processing Unit (GPU).