Catherine McCurry

Paths is an immersive 3D sound sculpture that a viewer performs by moving their body through space. The physical manifestation of the piece is an empty space inhabited by a distracted, and slowly wandering participant.  A white line along the floor suggests a possible navigation of the sculpture.

By navigating the sculpture, the viewer encounters a spatialized sonic landscape that is mapped to their physical location using a Kinect 3D camera and custom software written in openFrameworks. The combination of sounds at any spot in the sculpture is unique, as is every trip through it.

For the score assignment, I designed an instrument using the Kinect and OpenFrameworks. I arranged several sounds in a 3D space, conceptual mountains of sound. The volume of each sound is governed by the position of a performer within the space. The pan of each sound is controlled by the performer’s level (crouching and standing will cause the sound to move from left to right and back again).

I then designed a score, in the form of a path along the floor that a performer would walk along in order to perform the piece from mesh fabric strips. At every knot/bow in the score, the performer should crouch. At the end of the path, the performer should slowly back off of the mat.

The sonic content will probably change in the future. I will also probably tweak it so that the performer being too close, or accidentally exiting the sensing range of the Kinect will not cause the sound to cut out abruptly.

Week 1 – Slitscan Camera with Toby, Matt, and Fred

By taking a video with camera motion and then recombining various frame slices into a single image, we can simulate various camera positions and orientations that were not captured in the initial video.

source

Week 2 – Bounding box and centroid

source

Week 3 – Pointcloud color interaction

source

Week 4 – Experiments with mesh and shaders (mostly failed experiments)

source

source

I’ve been thinking about this project since last spring, and made a failed attempt last semester in ICM. When the Kinect came out, I knew it was a solution, which is why I’m in this class.

I worked on a production that adapted Samuel Delaney’s novel, Dhalgren, for the stage. One of my favorite images from the book was not realized in the play and it’s stuck with me as a design problem that I know there’s a solution to (see my final proposal from ICM last semester for more background).

In the book, members of the Scorpion gangs wear projector necklaces when they roam the streets at night. With the touch of a button, the necklaces produce a holographic animal that surrounds their body and identifies them – dragons, mammoths, unrecognizable blobs when they’re not working correctly.

I had previously envisioned a solid, neon colored animal shape for these shields, and thought of using skeleton tracking with OpenNI to animate a 3D character. I was nervous about the animated character, though, and pretty sure it would look dumb.

A simple, and I think effective, solution occurred to me late in the game. I reimagined the design of the holograms – they could be skinned as the creatures rather than shaped like them. I modified an example from class to remove the background, then map pixel information from existing images to the depth image from the Kinect. I projected this onto two layers of mesh that I stood behind, producing a faux 3D projection effect. I tried a couple images – two dinosaurs and a lizard.

Here’s a diagram of my setup:

This is a study for an effect to be used in a live performance.

source (this is also the code for the RGB experiments in the post above)