Using retrieved augmented generation for question answering with LLMs in the cybersecurity domain

Abstract

This work investigates the impact of Retrieved Augmented Generation on the performance of Large Language Models in a Question Answering scenario in the cybersecurity domain. 14 different context setups are built, based on the two data resources NIST and MITRE ATT&CK, and retrieved with tf-idf retrieval. The questions from Tihanyi et al. (2024) are prompted with and without context, and with and without answer options. The evaluation shows that the context only partially increases the performance of the three models meta.llama3-1-8b-instruct-v1:0, mistral.mixtral-8x7b-instruct-v0:1 and mistral.mistral-7b-instruct-v0:2. Further, there are differences observed in how well the models follow the prompt instructions, which clearly impacts the findings. In the future a more advanced retrieval method, different data resources for building the databases and a better alignment of the prompt length, could positively influence the results.