December 12, 2005
LIVE Video Tracking and Manipulation for Theatre Performance
Goal: To create a video-tracking tool for real-time video manipulation.
Context: Cross-media theatre. Specifically for performance where the performers are lit behind a scrim onto which live video is front-projected.
(see our website for visual examples)
How does it work? Two live cameras are used as input. One is an infrared camera which is used for tracking brightness. The other is a regular video camera that provides the live video content for display.
Control is shared between the video operator and the performer onstage. The video operator will turn the system on, and will select which mode is being used. The performer will wear an infrared light as a wearable prop, such as a ring or a bracelet. The infrared camera tracks the position of the light through space, and the display video responds to this movement. In this way, the performer can manipulate the live projected video in real time by moving her hand through space.
THE VIDEOS LINKED BELOW WERE ALL SHOT IN THE ITP LOUNGE USING BRIGHTNESS TRACKING
Mode 1: Rearranging. The performer can "grab" small sections of the video and move them across the image, and then "drop" them over another part of the video. The small sections that are moved remain live after they have been dropped. There is a choice as to whether an empty space is exposed where video has been "grabbed", or not. If there is space, this can be used by the performer as a window in the scrim to look through.
Mode 2: Audio response. The video responds in real time to audio input. The louder the audio input, the greater the fragmentation of the video.
Mode 3: Spotlight. The immediate area around the point that follows the performer's hand remains live video, while the rest of the image is frozen.
Questions:
Slow speed of image. The Processing core does not deal extremely well with live video processing. There were additional modes that I wrote which had to be left out because they slowed down the image too much.
Unreliability of video tracking. Video tracking requires a very controlled environment. Although it may well be possible to create a suitable environment in a theatre context, I wonder at what expense, in terms of lighting and video projection? Also, I doubt it would be worth it in terms of touring, as the amount of time it might take to get the tracking working in each new venue might be disproportionate to the value of the effect.
What does it really add, if the audience is not doing it themselves? The tracking has to be integrated into the aesthetic of the show, and watching the performer manipulate the video must be beautiful or relevant to content, or it just becomes demonstration of media technology.
Perhaps better to fake it? The audience does not care how you do it, as long as it looks right and adds to their experience?
Posted by edpurver at 03:04 AM | Comments (0)
November 21, 2005
Intro to Computational Media - Proposal for Final Project - Video Manipulation for Live Performance
Concept:
To reveal, move and manipulate both live and recorded video images using video tracking within the context of live theater... or possibly video chat.
Method:
The tracking object will be a wearable prop. So far I am thinking of a ring, on which I will fix several wide-angle surface-mount infrared emitters. The emitters will be powered by a small watch battery, which will be fixed to the underside of the ring. On the side of the ring will be a switch that responds to pressure, and which the performer can turn on and off by pressing her fingers together.
The set up will include two live cameras: an IR camera to track the wearable prop, and a regular video camera to provide live video images. The programming will be written using Processing.
I want the system to have shared control between a video operator and the performer. This is to lessen the burden on the performer and make the technical role of the performer as simple as possible. The operator will choose which mode of manipulation is being used, and the performer will apply that manipulation.
Initial ideas for manipulation:
1. The live video is broken up into fragments, and can be rearranged by movements of the performer's hand.
2. The performer can reveal an image by moving her hand through space.
3. The performer can conceal an image by moving her hand through space.
4. The performer can "write" or "paint" generated graphics onto the live video through movements of her hand.
Programming challenges:
1. To use keyboard input to switch between modes.
2. To make video tracking work in this context.
3. To effectively use Processing with two live video inputs.
4. All the graphical desires listed in the Initial Ideas...
Posted by edpurver at 07:03 PM | Comments (0)
Computational Media - Parsing data
With an assignment to parse data from either a local or an online data source, I decided to try both and to work with text. I tried two simple experiments. The first was to take the text of Tony Blair's recent speech to the Labour Party conference and to search it for three individual keywords. The full content of his speech is printed in the background, while each instance of the keywords is printed in the foreground.
Here's a screen shot of the program running.
The second was to perform a crude parsing of the database of civilian casualties listed by the website iraqbodycount.org.
Posted by edpurver at 03:45 AM | Comments (0)
October 31, 2005
Intro to Computational Media - live video tool
For my midterm for the Intro to Computational Media class, I decided to use Processing to create a live performance tool. As my collaborator and partner, Sara Kraft, is a singer, and as we use a lot of video in our performances, I decided to try and create a tool that would visualize live sound input through the medium of live video.
Steps:
*Figure out how to break a live video capture into independent sections, that can either sit together to display the whole image, or move around to break up the image
*Use live sound input in Processing for the first time.
*Make the video respond to the sound.
Following Dan Shiffman's advice, the way I chose to do this was to use the copy() function to copy areas of the live video capture into Pimage variables. So, the first stage was to simply try to use the copy function. I tried it first with a still image, using the random function to prove that the image fragments could move independently of each other.
Here's a link to the applet.
The next stage was to add the sound input. I decided to stay simple and just have the image respond to fluctuations in sound input volume. So, the louder the singer sings, the more the image fragments.
Then the next stage was to start using live video capture as opposed to using a still image. To begin with I continued to just divide the image into 4 equal quarters.
Here is a link to the code.
Here's a screen shot of the video responding to some vocal sound:
The final stage was to add a little bit of complexity to the presentation of the video. I broke it up into smaller images, that sit on top of a background layer. The background layer is the same video, but displayed whole, and with an Invert filter. Instead of all moving offscreen towards the corners of the display, the image fragments now move in different directions, but should still clear the screen if the singer sings loud enough, leaving an uninterrupted view of the background layer of video.
Here is a link to the code.
Here's a screen shot of the video in silence (yes, I'm just sitting there with my mouth open):
Here's a screen shot of the video responding to vocal input (this time I'm singing (badly)). You can see how the video fragments are moving across and off the screen in response to the sound input level.
Posted by edpurver at 05:53 PM | Comments (0)