ITP Thesis Archive 2025

Project Description

Learning to play an instrument can be an intimidating challenge for a beginner, with several barriers to entry that may dissuade players from advancing enough to truly enjoy the hobby. One way to inspire people to begin this journey is to give them a taste of the end result, and help develop an appreciation for how fun making and playing music can be by removing those barriers and allowing them to play music easily and seamlessly without any prior knowledge.

Hover Craft is a physical interface for music creation, wherein users hover their hands over various components of the device to play notes and modulate parameters of the sound. Slide-out tabs built into the device explain these modulatable elements - the basic musical concepts of scales, chords, and arpeggios - and provide instructions on how to use the device. By utilizing machine vision, a standard webcam mounted above the interface tracks users hands, enabling a novel touchless experience akin to playing an air-piano.

The available notes to play are locked to scale, meaning that no matter what the user does, it’s impossible to play “out of key”, creating a rewarding experience that can enable people of all ages to gain an appreciation for the art of playing and creating music.

Technical Details

A webcam outfitted above the interface sends a video feed to MediaPipe, which uses its machine learning model to track skeleton joints in the hand. The user’s index finger tips are tracked, and when they hover over certain UI elements on the digital interface, two things happen simultaneously in custom software built in TouchDesigner by the artist:

1) MIDI note data is sent to Ableton Live to play music notes
2) A trigger activates a light pattern corresponding to the note which is being hovered - video signal of this activation is sent to MadMapper to light up the LEDs under the frosted "pucks" on the interface.

Hovering over the left hand panel activates a short timer that flips between different internal switches of modulatable parameters. When this switch is triggered, it sends MIDI control data to Ableton live to switch between various effects chains (instrument, scale, chord, arpeggio). In total, there are 64 unique combinations of these parameters.

Users can choose between:

- 4 Instruments (Piano, Woodwind, Strings, Mallet)
- 4 Scales (Major, Minor, Harmonic Minor, Phrygian Dominant)
- 2 Chord Options (Single notes or 3 note chord, with the root note, 3rd, and 5th scale degrees)
- 2 Arpeggiation Options (Single notes or arpeggiated sequence at 1/16th notes, 3 steps jumping a 3rd note per step)

--

For the fabrication, I aimed to create a highly portable device that could be easily assembled and installed anywhere with as little extra equipment as possible. Hover Craft is built from several layers of laser cut wood with an acrylic topping - this layer based design allowed for the installation of electronics such as the LED circles for the 8 acrylic "pucks", the LED matrix for the modulator panel, and magnetic electronics connectors built inside the casing. An acrylic tube holds up the webcam with a 3d printed mount to keep it oriented exactly in the center of the device, and the holder for these tubes was designed with "teeth" that slide into the two halves of the device and lock them together.

Slide-out acrylic "cards" inform the user of the different musical concepts throughout and explain the abstract icons etched into transparent acrylic on the control panel. A larger acrylic card on the right has etched instructions that inform the user on how to interact with the device.

Research/Context

A motivating factor for making this piece was investigating how children are first exposed to music. Imagine a 5-year-old child placed in front of a Grand Piano - their first intuition is likely banging on all keys simultaneously, creating a cacophony that doesn't inspire continued learning.

Now imagine a custom piano where any combination of notes sounds harmonious. There's a good chance that the child would be more likely to continue on the path of learning the instrument, having experienced that feeling of "making something sound good" instantaneously.

Giving people a taste of how it feels like to play well-designed music may give them an appreciation for how enjoyable this hobby can be, providing an achievable goal as they begin their journey.

This idea stems from a key concept from video game design: A "power fantasy" or "taste of power" sequence is where players get to experience a fully powered version of their character before being reset to start the progression journey. These terms refer to those situations where at the start of the game, players are temporarily given supercharged abilities to give them a sneak peek at what is to come throughout their playing experience. This might exhibit itself like having full health, the most powerful weapons, or other abilities that players might have to develop over the course of the game and is a good way to teach some basic mechanics that they’ll need later on.

This technique is widely recognized as an effective motivational tool in game design - By giving players that initial taste of power, designers give players a goal to look forward to and the confidence that they’ll eventually be rewarded by struggling through the early stages of the game.

By giving players of Hover Craft the ability to immediately play music that is in key and sounds good, it gives them this same "taste of power" that may inspire them to struggle through the early stages of learning an instrument.

---

On the design side, I was primarily inspired by abstract instruments such as those found in the annual New Interfaces for Musical Expression conferences, discovering novel ways to leverage technology to create a rarely seen input style for playing music. Another key factor was that, as much as possible given the technical limitations available, I wanted to implement the concept of game “juice” - aka the haptic, audio, and visual cues that add small moments of delight into your playing experience.

I took heavy inspiration from children’s toys like LEGO to create a significantly more portable version of the device by fabricating it in a completely disassemblable manner. This LEGO-like design allowed for the built-in instruction manuals on the left and right hand sides that allow for the user to learn how to use the device and some key facts about the different modulation options.

Thesis Advisor: Tiri Kananuruk
Thesis Resident: Josh Zhong
Special thanks to the following people who were especially hands-on in helping me develop this: Aaron Montoya, David Rios, Gabe Warshaw, Ian Cox, Jasmine Nackash, Justine Chung, Matthew Lau, Octavio Figueroa, Paulina Olivares, Pedro Galvao Cesar de Oliveira, Phil Caridi, Proud Aiemruksa, Steven Jos Phan, Torin Blankensmith & Dom Scott, and the ITP Community as a whole.

---

If this project interests you and you want to collaborate on something similar, feel free to reach out!

andrelira.design@gmail.com
http://instagram.com/doctajeep
http://linkedin.com/in/alira