Application of fuzz testing for testing highly configurable systems

Abstract

In modern software development, particularly in the automotive industry, Highly Configurable Systems (HCS) are frequently utilized to create various software versions. These systems present their own set of challenges due to the multitude of configuration options available. A defining characteristic of HCS is their ability to generate multiple instances through configuration, enhancing feature reuse. In the automotive sector, customized and reliable software is essential for different vehicle models and functionalities. Additionally, managing the time and effort required for thorough testing without sacrificing quality is essential. This study emphasizes the importance of safety, security, and quality in Highly Configurable Systems (HCS). Fuzz testing is a valuable quality assurance tool for identifying and eliminating potential vulnerabilities, though it can be time-consuming and resource-intensive for highly configurable with frequent feature changes. We introduced a methodology for testing HCSs using fuzz testing, focusing on testing different versions and variants of cars. By concentrating on the parts impacted by changes, we aimed to optimize resources and streamline defect identification. To achieve this, we used Continuous Integration and Continuous Deployment (CI/CD) systems, directing testing resources to the most recently changed code. This optimization expedites development while maintaining system integrity and quality. In our research, we used two Highly Configurable Systems (HCS) projects with fuzz tests and a Continuous Integration (CI) system, utilizing pull requests to simulate frequent changes. Our method ensured that only recent updates were tested. Coverage reports from fuzz testing were used to map test cases, improving transparency and effectiveness across the system. Deploying a CI system optimized the fuzzing process in HCS, reducing time spent on redundant tests for unmodified code and detecting issues such as memory leaks and undefined behavior. Additionally, coverage reports provided measurements of feature testing coverage, further enhancing the efficiency and effectiveness of our approach.