Active learning strategies for deep learning based question answering models

Abstract

Question Answering (QA) systems enable machines to understand human language, requiring robust training on related datasets. Nonetheless, large, high-quality datasets are only sometimes available due to cost restrictions. Active learning (AL) addresses this challenge by selecting the data with high information value as small subsets for model training, considering computational resources while preserving performance. There are many different ways to detect the information value of the data, which in turn leads to a variety of AL strategies. In this study, we aim to investigate the performance change of the QA system after applying various AL strategies. In addition, we use the BatchBALD strategy, compared with its predecessor, the BALD strategy, to inspect the advantages of batch querying in data selection. Eventually, we propose Unique Context Selection (UC) and Unique Embedding Selection Methods (UE) to enhance the sampling effectiveness by ensuring maximal diversity of context and embedding within querying samples, respectively. Observing the experimental results, we learn that each dataset has its own AL strategy that brings out its best results, and there is no universal optimal AL strategy for QA tasks. BatchBALD maintains the modeling results similar to BALD in the regular setting while significantly reducing computation time, though this feature is not practiced in the low-resource setting. Finally, UC could not enhance the effectiveness of AL since half of the datasets used in this study consisted of more than 65% unique contexts. However, the effect of UE enhancement deviates across datasets and AL strategies, but it can be observed that most of the AL strategies with the best effect of UE enhancement can increase by more than 0.5% F1. Compared with context, a feature of datasets is limited to natural language processing tasks; embedding is more generalized and has a good enhancement effect, which is worth studying in depth.