Augmented reality feedback visualization for guitars in music education

Abstract

This paper presents an application that helps users to learn play the guitar. To be able to give feedback, the physical guitar is equipped with markers and we track every note that is played. Subsequently, visualizations can show the user if and at what point they made a mistake while playing. A learner can see the visualizations either on a screen or through AR glasses. Additionally, overview graphs of whole songs can be displayed that indicate in bars, pie charts or other geometries how often the respective notes were played correctly or incorrectly. Likewise, multiple visualizations can be displayed simultaneously to compare the different times the same song was played by a user. Our implementation can be used with the Microsoft HoloLens or with monitor and webcam. %, but since it is a unity project, it can easily be built for many other devices as well. Our results with monitor and webcam showed good working and understandable results. For our evaluation we used case studies. If our program can improve learning to play the guitar has to be shown in representative user studies. Due to the SARS-CoV-2 pandemic we had only a short time to use the Microsoft HoloLens. For this reason we could only show that our visualizations would work but we found no solution to read the .JSON files that we used as input. Therefore we were not able to evaluate our application with the Microsoft HoloLens.