AxAR: An Augmented Reality Guitar Learning Tool
Orpheas Kofinakos
Advisor: Kari Love
This project performs research in the area of alternative music learning tools within XR, taking into account and eliminating the cost factors of additional hardware that have been used in similar previous work.
Abstract
The app AxAR is an Augmented Reality application for independently learning songs on the guitar. This project focuses on providing the user with real-time guidance and feedback while playing the instrument, using a Head Mounted Display with XR capabilities. It provides a more intuitive and seamless visual approach to music learning compared to traditional methods, which rely on shifting between different points of view between the instrument and music instructions. Additionally, it provides real-time performance feedback to the user without a comprehensive setup.
Technical Details
Tech Stack:
Unity
C#
Meta Quest 3
Meta XR SDK
Research tech:
OpenXR
CREPE
SwiftUI
ARKit
Research/Context
The integration of AR technologies in guitar playing has been a topic of research for more than two decades, using a variety of forms, most of them relying on a digital mirrored image of the guitar player where instructional elements can be added.
The recent advancements in accessibility and performance of headset technologies have democratized a number of virtual user interactions previously impossible. Some of these aspects offer a fertile environment for the development of guitar learning software that surpass the constraints of screen-based devices, namely the focus shift of the player’s point of view.
This thesis concentrates on providing of a new user interface appropriate for guitar learning in XR, and methods of evaluating user performance using hand tracking and audio analysis with no additional hardware besides the headset device itself.
The existence of similar research was a push towards elaborating on the user experience of such an application, refining the concept of serially overlaying graphics on a guitar fretboard through a novel iteration. Part of this project also dealt with ways that suggested components for AR music notation can be positioned in the surrounding space of the fretboard rather than on top of it, along with conventional GUIs that are essential in the learning of a song on the guitar.
Further Reading
This project is partly inspired by music games such as Guitar Hero and Rocksmith, as well as personal experiences of learning the instrument. The implementation of HMD-based guitar learning is predated by at least two works in this specific form (GuitXR (2021), AR Hero (2023)). Previous research noted technical difficulties in their approaches which provided guidance in some of the methodologies followed. Additionally, past research has resolved in using additional gear other than the headset, which may reduce the accessibility of the tool. One of the main goals of this project is to create a learning experience that is intuitive and accessible, a compact form is therefore desirable where there are little to no components besides the AR hardware. Existing music learning AR apps for different instruments, such as Pianovision, were also part of the research process.
