For the past few weeks I’ve been exploring the relationship between motion and sound through the medium of dance, with the intention of creating a more meaningful link between the visual and sonic aspects of the performance. Historically choreography has been grafted onto existing pieces of music that were written in a completely separate context, or music was scored with the choreography in mind but was still performed by musicians, not dancers. In many cases a conductor has served the role as a link between the two. I can’t necessarily argue that the repertoire has suffered because of this disconnect, but I though it would be worthwhile to try to eliminate the middleman and tether the score directly to the dancers. The following improvisations contain sound elements that react to the motion of the dancer’s body in real time, generating the score on the spot. I was fortunate to work with the super talented dancer Jennifer Landa on this project. My apologies for the focus issues in the video.
The biggest challenge I faced was the lack of rehearsal time with Jen. There’s only so much you can predict without actually getting in a room with the performer and testing. I had a couple weeks to work before our first rehearsal so I wanted to get the entire architecture done beforehand, which would leave me with another week to tweak it before our final shoot. First I set about putting my sounds together. I decided to create three layers of sound. One layer would be a drone that remains constant and is not controlled by the dancer in any way. For this I chose a low, rumbly, drone which provides some substance but leaves plenty of room for other sounds to skate above it. Second is a layer of sound that responds to motion in general. Here I used some filters without any audio going through them. By manipulating the parameters of the filters I was able to create some glitches that I find very appealing. In the recording you can hear this layer as the ghosty pitches and watery glitches. The position of Jen’s feet are controlling this layer. The third layer of sound are samples that are triggered by specific gestures. For this I’m using some bleeps and a synth stab. The final audio component is a system that mimics the sound of electricity. I wanted to have the electricity gain intensity as two dancers moved toward each other. Unfortunately I couldn’t get a second dancer so instead I set a hotspot, in the video toward the right front of the stage. This starts a couple of minutes into the video.
In the future I would like to incorporate some projections into this project, so with that in mind I am using Processing to take in the skeleton data from the Kinect. From there I send OSC messages to Pd, where the audio is generated based on the input. I built a series of sub patches to parse the OSC messages, which make it easy for me to nimbly switch the data being sent in Processing and interpret and route the data in Pd. This flexibility was especially important because I knew there was going to be a lot of trial and error and not a lot of time when I got in the studio with Jen. The initial iteration mapped hand position to the filter layer and used a couple of hand gestures to trigger the samples.
A big challenge in the first rehearsal was calibrating to the space and to Jen’s body. Up until that time I had been testing in my cramped apartment and in cramped rooms at school. Also, it became apparent pretty quickly that Jen’s movements were completely different from the mine. I had never thought about rolling around on the ground for instance. The Kinect had some trouble keeping track of her. I think it was great for Jen to get some time working with the setup because it gave her ideas as far as what types of movement would be easy for the Kinect to interpret. Generally, the slower she moved the more reliable it was. And she started learning how to prevent inadvertent triggers of the samples. At the end of the rehearsal we decided to switch the filtered layer of sound to her legs because she tends to focus a lot of her moves there. We were also able to test the electricity layer in a basic way, but there was still some work to do mapping that data in an effective way.
In the week between the rehearsal and the final shoot I worked on expanding the system for multiple dancers, increasing reliability, fine tuning my data mapping, and developing the sounds. I was pushing the system to the limit so I had to pick and choose what data to send at any given time. The day of the shoot we spent some time testing and fine tuning. I could see how Jen’s level of control improved as time went on. For the final take Jen went non-stop for 45 minutes! The Kinect lost her a couple times and my filters exploded a few times too (I tried using a limiter, but it wasn’t fool proof). I plan on editing the footage some more and putting up more excerpts.