Experimental investigation of the consequences of expected source code understandability

Abstract

Understanding program code represents an essential part of most developers’ work. Any maintenance task requires the comprehension of the corresponding code as a first step. For that reason, software companies pay close attention to the quality of their codebase. It has become a standard to incorporate static analysis tools in the software process in order to automatically identify code smells and help developers to improve their code. However, the majority of metrics that are used in static analysis tools lack empirical evidence. We do not know how these unvalidated metrics influence the cognitive process of developers in regard to program comprehension. In this work, we investigate the consequences of presenting different understandability values to developers prior to them inspecting a code snippet. We analyze to what extent this understandability metric impacts the expectations, motivation, and affective states of programmers. To this end, we conduct an experiment through an online survey with 81 developers randomly assigned to one of three treatment groups with different presented understandability values. Before and after the task of judging the understandability of a code snippet, participants have to report their expectations, motivation, and affective state with regard to understanding the code snippet. In addition, two code snippets are used to evaluate differences in perception, motivation, affect, and understandability judgment as a result of the actual difficulty of the code snippets. Our findings show no significant effect for expectations, motivation nor affective states as a consequence of the presented understandability value. However, we observe a significant positive linear relationship with expectations explaining 18.3% of the variance of motivation at an alpha level of 0.0056 with a large effect. Similarly, differences between expectations of understanding the code snippet before seeing it and the perception of understanding it afterwards demonstrate the same significant positive relationship with motivation difference. Our results show an even larger correlation between expectation to perception difference with a happiness difference of participants, indicating that being positively surprised by understanding a code snippet corresponds with increased motivation and happiness. Lastly, presenting programmers’ different understandability values does not influence the assessment of the code snippet. Generalization of these results is limited by the use of small code snippets of 20 to 30 lines of code. Furthermore, expectation and motivation are measured through a self-created and therefore unvalidated instrument. The results showcase the importance of managing expectations in order to increase motivation and affect of developers. Additionally, contrasting to prior work understandability metrics seem to not pose a threat of biasing programmers in their expectations towards and assessment of source code.