watch what you are

Justin Downs
Rucyl Mills

Watch What you Are is a immersive computer vision system that links the 2-D virtual world of movies and tv to the 3-D present of the end user.


Classes Live Image Processing and Performance

Watch What You Are, is a vision tracking system that looks for visual pattern matches between a movie or any other two dimensional source, and a real time video feed. The visual image comparison we are focusing on, is between the body positions of actors and actresses in the movies, and the positions of the games participants. Using this comparison a link between the two dimensionally represented reality of the actors and the current reality of the participants is created.

Game overview:
When a person enters into the camera's field of focus the beginning film triggers (in this case a 70's kung fu movie) the opening scene and asks the participant to choose a difficulty setting by putting their hand over the word "easy", "medium" or "hard". Once the level is selected the movie starts to play, it continues on till a “break point” occurs where the participant is challenge to mimic the actor/actresses position. In the challenge they receive a certain amount of time (based on the difficulty level) to try and mimic the body position of the actor/actress, With a successful positioning the movie continues to play on till the next challenge, if all the challenges are met you view the whole movie and see the victory credits. If you fail a challenge the end credits run immediately preventing you from seeing the rest of the movie.

Technology used:
The overall program flow is done in Max/Msp to easily manipulate the video clips, with the video tracking Max objects custom written using java (C++ in the next version for speed). The setup consists of a screen, camera, computer and white background.

Most of the time was trying to figure a way to make a vision tracking system that would. Work well in " memorizing" and comparing bodily shapes.

People that have watched or seen movies.

User Scenario
We would like to have a venue big enough to implement the project using a full sized movie screen

Thought on why we think and how we look.
Just as the orangutan populations of Indonesia spread their knowledge of survival through show and tell (Why Are Some Animals So Smart? The unusual behavior of orangutans in a Sumatran swamp suggests a surprising answer. By Carel van Schaik)(scientific american 2006) , the human brain also is trained to mimic and learn through visual cognition, we watch then do.

With the advent of 2 dimensional visual cognition during the Renaissance (primarily in northern Italy) a new logic of interpretation came into existence (Kozbelt,Aaron.
Psychological Implications of the History of Realistic Depiction: Ancient Greece, Renaissance Italy and CGI
Leonardo - Volume 39, Number 2, April 2006, pp. 139-144
). Instead of a direct 3 dimensional mimicry which pairs the sense of touch to visual cognition a new sense of time replaces the phantom limb of touch in 2 dimensional space. This substitution creates a time lagged reality, which allows for the broader interpretation of self. The self of the future or past is highlighted in this form of learning at both the cognitive and meta-cognitive (personal storyboarding approach to reality) levels.

With the visual learning processes of the orangutan the mimicry and success or failure of that action is immediate. Whereas the replication of 2 dimensional imagery only happens out of time. Since the user is displaced from an immediate mimicry of the visual images through the media of time, they have to surrogate themselves to use the real tools the actors or actresses are manipulating in their performance. This can lead to unrealistic emotional expectations or maybe a permanent sense of almost completion.

The relation of verbal imagery is different, although story telling has a gestrual structuring that accompanies all auditory language which possess the same dissonance of time based learning. With the effects of verbal reproduction the hardwired cognitive aspect of learning is provided by the user in a direct relationship of self. So the tools and person exists in the same realm of now because both are created from that instance.

So what happens? is TV bad should we never draw pictures? Some religions that will go unnamed say yes and have stuck with the ever-removed visual/language interpretational mechanisms that hide behind the eyes of the viewer. We who have grown up in the post war Europe, the 1980s Japan and the ubiquitous America have been lead down another route, that of supreme images as supreme context. The battles of our common day super egos are played out in the ticket sales of cinematic titans. We are larger than life from the beginning we are great in our future tragic in our past and mediocre in our present. We have learned to see through screens with screens for screens. This has allowed us to do amazing feats of bravery and reach dizzying heights of logic.

We now can know the effects of actions and the structural equations of thousands of places on the earth all at once through the glimpse of a talking six o’clock news head. With the help of a Bollywood movie we can pick out villains from a crowd and predict the future of these 40 foot titans in the screen, we know they will be killed when they open the door and we just watch it happen, let it happen. We sit and watch thousands of people get murdered every year even though we have the foresight and the knowledge to stop it. We could turn the TV off right before they get killed, and if we all did it all in unison maybe Laura palmer would still be alive.

But we watch and learn and act although we don’t actually mimic, even though we learn the emotions to live a thousand deaths and jump from an airplane with only a jet pack we never do. We try and try hard though, using all the tools we know how to use. But since the learned knowledge was never actually practiced we fall short and move to New York, by an expensive track bike and imitate something in three dimensions thinking it will change you. But again unfortunately most are too far gone too ever carry anything but surrogate emotional content of a future self who never comes to hang out.

So this project is not only a game but also a way to reassert the connection between seeing and doing. To show and feel what we are capable of and how we can use it so our stories will not die with us and the next generation will not just be hollow vessels of these Titians but actually will be Rambo’s of the first blood order.