Context: Cross-media theatre. Specifically for performance where the performers are lit behind a scrim onto which live video is front-projected.

(see our website for visual examples)

How does it work? Two live cameras are used as input. One is an infrared camera which is used for tracking brightness. The other is a regular video camera that provides the live video content for display.

Control is shared between the video operator and the performer onstage. The video operator will turn the system on, and will select which mode is being used. The performer will wear an infrared light as a wearable prop, such as a ring or a bracelet. The infrared camera tracks the position of the light through space, and the display video responds to this movement. In this way, the performer can manipulate the live projected video in real time by moving her hand through space.

LINK TO THE CODE PT 1

LINK TO THE CODE PT 2

THE VIDEOS LINKED BELOW WERE ALL SHOT IN THE ITP LOUNGE USING BRIGHTNESS TRACKING

Mode 1: Rearranging. The performer can "grab" small sections of the video and move them across the image, and then "drop" them over another part of the video. The small sections that are moved remain live after they have been dropped. There is a choice as to whether an empty space is exposed where video has been "grabbed", or not. If there is space, this can be used by the performer as a window in the scrim to look through.

CLICK HERE TO WATCH VIDEO

Mode 2: Audio response. The video responds in real time to audio input. The louder the audio input, the greater the fragmentation of the video.

CLICK HERE TO WATCH VIDEO

Mode 3: Spotlight. The immediate area around the point that follows the performer's hand remains live video, while the rest of the image is frozen.

CLICK HERE TO WATCH VIDEO

Questions:

Slow speed of image. The Processing core does not deal extremely well with live video processing. There were additional modes that I wrote which had to be left out because they slowed down the image too much.

Unreliability of video tracking. Video tracking requires a very controlled environment. Although it may well be possible to create a suitable environment in a theatre context, I wonder at what expense, in terms of lighting and video projection? Also, I doubt it would be worth it in terms of touring, as the amount of time it might take to get the tracking working in each new venue might be disproportionate to the value of the effect.

What does it really add, if the audience is not doing it themselves? The tracking has to be integrated into the aesthetic of the show, and watching the performer manipulate the video must be beautiful or relevant to content, or it just becomes demonstration of media technology.

Perhaps better to fake it? The audience does not care how you do it, as long as it looks right and adds to their experience?

Method:
The tracking object will be a wearable prop. So far I am thinking of a ring, on which I will fix several wide-angle surface-mount infrared emitters. The emitters will be powered by a small watch battery, which will be fixed to the underside of the ring. On the side of the ring will be a switch that responds to pressure, and which the performer can turn on and off by pressing her fingers together.

The set up will include two live cameras: an IR camera to track the wearable prop, and a regular video camera to provide live video images. The programming will be written using Processing.

I want the system to have shared control between a video operator and the performer. This is to lessen the burden on the performer and make the technical role of the performer as simple as possible. The operator will choose which mode of manipulation is being used, and the performer will apply that manipulation.

Initial ideas for manipulation:
1. The live video is broken up into fragments, and can be rearranged by movements of the performer's hand.
2. The performer can reveal an image by moving her hand through space.
3. The performer can conceal an image by moving her hand through space.
4. The performer can "write" or "paint" generated graphics onto the live video through movements of her hand.

Programming challenges:
1. To use keyboard input to switch between modes.
2. To make video tracking work in this context.
3. To effectively use Processing with two live video inputs.
4. All the graphical desires listed in the Initial Ideas...

Here's a screen shot of the program running.

CLICK HERE TO RUN THE PROGRAM

The second was to perform a crude parsing of the database of civilian casualties listed by the website iraqbodycount.org.

CLICK HERE TO RUN THE PROGRAM

Aim:
The Light Window is an interactive light fixture, designed for rooms that have low levels of natural light. Many rooms in urban centers such as Manhattan have very little natural light. Windows are often small, and often face onto a neighboring building, meaning that direct sunlight is rarely, if ever, able to shine through.
The aim of the Light Window is to provide the ambient light that is missing from such living spaces. Through an interactive interface, the user is able to adjust the level of light that is emitted from the window, and also to select from five different ambient soundtracks. The aim is to give the user the ability to change and improve her living environment.

Click here to watch video

The Process:
We considered how opening a window affects change in a room, and noted changes in sound, temperature, smell and point of view. We decided that the most important attribute of a window was its capacity to bridge an internal environment with an external environment. Through opening to the outside, the inside is temporarily changed.

We decided that we wanted our device to reflect some kind of change in environment, and we identified light and sound as the two most key elements that can be variable in a room.

The Interface:
The Light Window is controlled through moving the wooden slats that cross the face of the window. Its design reflects both contemporary and classic design. The wooden controllers cross the window pane, reminiscent of a classic window frame. However, instead of being static, structural components, the wooden controllers are adjustable, and the horizontal slat controls the level of light, and the vertical slat controls the background soundtrack.

The Components:
The Light Window takes information from IR rangers to track the position of the wooden controllers. This information is processed through a microcontroller, converted to AC current, and then used to dim and brighten a series of Full Spectrum light bulbs. A second microcontroller takes information from the vertical controller and using a max/MSP patch, selects a soundtrack depending on the position of the slat. In this initial prototype, the audio was played over external speakers, but in a finished version, the speakers would be embedded into the design of the window.

Our recent assignment in the Spatial Design class was to build a labyrinth. Inspired by the recent class readings by Jorge Luis Borges (from the Labyrinths collection), I decided to work with words. Borges seems to be interested in how our "reality" is fluid and flexible, and always shifting, depending on our understanding of it.

I have been interested in moments of reality breakdown for some years now, and I decided to make a conceptual labyrinth, based on miscommunication. It sometimes makes for fascinating viewing in classes, when someone asks a question which is misunderstood or misheard, and a long nonsensical answer is given in response. Due to the politeness of our social conduct, the answer is rarely interrupted, and then the answer itself provides a new subject for conversation, and is questioned on its own terms and the conversation takes an entirely new and unlogical direction.

So, I am interested in the possibility for two people to take two different paths within a conversation. At first they are related, but then they become more and more distant and unrelated until they are in different conversations, but still speaking together.

Thanks to my brother, Matt Purver, I got online access to a large database of conversations recorded in England during the 90's. These conversations were recorded and transcribed as part of a project by the Logic, Language and Computation Group at King's College, London, where Matt did his PhD before going to work at Stanford. I spent some hours reading through the conversations, and especially the section which isolated moments of misunderstanding.

I selected one particular excerpt of dialogue as an example of confused communication. It's between two schoolboys in England, and I posted it on the site I made to present my ideas for my labyrinth in class, here

I evisioned two different way to structurally communicate the kind of labyrinth that I am interested in. One was an installation where at the start of the dialogue the two speakers are in the same room, the walls of which are made of a transparent material. With each misunderstanding the speakers are moved into separate adjoining rooms, so that they get further and further away as the conversation gets more confused. The walls are all thin and transparent, so they can still see and hear each other, its just that the quality of sound and vision degrades with each degree of separation, so that eventually they will only see each other as blurred shapes and hear each other as vague sounds.

The second was a room that structurally changes itself with each misunderstanding. The structure begins as a very recognisable, straightforward room, but then changes, bit by bit, into a more disorienting and unrecognisable environment. I put simple flash animations on the site to convey my thoughts in class.

Then I began to build a small model, out of transparent, slightly frosted plastic. As I built it, it started to become a three dimensional maze, with two points of entry through the same initial chamber. The two paths then run around, under and over each other, but never meet again. It could be used as a game, with ball bearings inside the labyrinth, and the object of the game is to pass both balls through the labyrinth together.

However, I didn't feel that this model was conveying the feeling of disconnection that I wanted to portray. So, I decided to make a video, to try and express the labyrinth of sound and vision that we are living in all the time. Myself and Andrew Schnieder recorded the excerpt of dialogue from the English schoolboys, and then I recorded images and sounds from around ITP and the surrounding streets. What results is a strange, dreamlike labyrinth of sound and light.

Click here to watch the video

Here are some stills from the video

Steps:
*Figure out how to break a live video capture into independent sections, that can either sit together to display the whole image, or move around to break up the image
*Use live sound input in Processing for the first time.
*Make the video respond to the sound.

Following Dan Shiffman's advice, the way I chose to do this was to use the copy() function to copy areas of the live video capture into Pimage variables. So, the first stage was to simply try to use the copy function. I tried it first with a still image, using the random function to prove that the image fragments could move independently of each other.

Here's a link to the applet.

The next stage was to add the sound input. I decided to stay simple and just have the image respond to fluctuations in sound input volume. So, the louder the singer sings, the more the image fragments.

Then the next stage was to start using live video capture as opposed to using a still image. To begin with I continued to just divide the image into 4 equal quarters.

Here is a link to the code.

Here's a screen shot of the video responding to some vocal sound:

The final stage was to add a little bit of complexity to the presentation of the video. I broke it up into smaller images, that sit on top of a background layer. The background layer is the same video, but displayed whole, and with an Invert filter. Instead of all moving offscreen towards the corners of the display, the image fragments now move in different directions, but should still clear the screen if the singer sings loud enough, leaving an uninterrupted view of the background layer of video.

Here is a link to the code.

Here's a screen shot of the video in silence (yes, I'm just sitting there with my mouth open):

Here's a screen shot of the video responding to vocal input (this time I'm singing (badly)). You can see how the video fragments are moving across and off the screen in response to the sound input level.

Here is a photo:

Then, as a next stage in the investigation, I created this simple shape using wire and pantyhose. The shape was created by the movements through space of my hands and feet during the leap. The photo below shows the shape turned vertical.

Here are some video stills of what I made.