Collective variables in data-centric neural network training

Abstract

Neural Networks have become beneficial tools for physics research. While they provide a powerful tool for data-driven modeling, their success is accompanied by a lack of interpretability. This thesis aims to add transparency to the opaque nature of NNs by means of collective variables, a concept well-known in the field of statistical physics. Three collective variables are introduced that emerge from the interactions between neurons and data. These observables enable one to capture holistic behavior of the network and are used to conduct an analysis of neural network training, focusing on data. Through the investigations, the collective variables are applied to selections from a novel sampling method: Random Network Distillation (RND). Besides studying collective variables, the investigation of Random Network Distillation as a data selection method composes the second part of this thesis. The method is analyzed and optimized with respect to its components, aiming to understand and improve the data selection process. It is shown that RND can be used to select data sets that are beneficial for neural network training, giving rise to its application in fields like active learning. The collective variables are leveraged to further investigate the selection method and its effect on neural network training, revealing previously unknown properties of RND-selected data sets. The potential of the collective variables is demonstrated and discussed from a data-centric perspective. They are shown to be discriminative towards the information content of data and give rise to novel insights into the nature of neural network training. In addition to fundamental research on neural networks, the collective variables offer several potential applications including the identification of adversarial attacks and facilitating neural architecture search.