A systematic review of explainable AI methods for transformers in software engineering

Abstract

BACKGROUND: Transformer-based models have revolutionized artificial intelligence (AI) applications in software engineering (SE), supporting tasks such as code generation, debugging, and documentation. However, their opaque decision-making processes limit trust and usability. Explainable AI (XAI) methods aim to address these issues by improving interpretability, but their integration into transformers for SE remains underexplored. Existing XAI techniques face challenges in aligning with developer workflows and adapting to complex transformer architectures. OBJECTIVE: This study aims to provide an overview of the current state of research on XAI methods for transformer models in SE, focusing on their applicability, limitations, and empirical evidence supporting their effectiveness. By identifying key challenges and research gaps, the study explores how XAI can be better integrated into SE workflows to enhance transparency, usability, and trust in AI-driven tasks. METHODS: Following the guidelines by Kitchenham and Charters, a systematic literature review was conducted using IEEE Xplore, ACM Digital Library, and Google Scholar. A total of 32 studies were selected through automated searches and snowballing. Research questions were formulated to evaluate existing XAI methods, their suitability for large language models (LLMs), and their impact on SE tasks. RESULTS: Attention-based, gradient-based, and model-agnostic techniques were identified as key categories of XAI methods. The SHapley Additive exPlanations (SHAP) method emerged as the most empirically supported approach for LLMs in SE. However, limitations in scalability, usability, and interpretability persist across methods. Significant gaps were also identified in the application of XAI methods to transformer-based AI models, including dataset biases, the lack of standardized evaluation metrics, and insufficient practical validation. CONCLUSION: XAI methods hold great potential for enhancing transparency, trust, and productivity in AI-driven SE tasks. However, their real-world application is constrained by practical challenges and theoretical gaps. Future research should prioritize empirical studies, user-centric designs, and scalable methods tailored to transformer models in SE. Structured evaluation metrics and benchmarks are essential for advancing the field.