For my computational cameras final project I originally had a grandiose plan of an elaborate Augmented Reality application for the iPhone that would allow me to create animated cartoon-like worlds over the surrounding New York City landscape. Originally I had thought that because of a brief foray into OpenFrameworks and C++ I would be able to easily compile a proof of concept involving Broadway in front of the Tisch Building and a rolling pool ball. As I pushed forward with this idea, it quickly became clear that I was not going to learn OpenFrameworks (or Objective C) in the mere 2-3 weeks before the project was due. The next step was to return to using one of many AR platforms already out there to place my idea into the space.
I had attempted to work with Junaio a bit for a previous project, and again I found myself fighting with the 3d formats. While I was able to upload stationary 3d objects within the .obj file, these files don’t allow encapsulation of animation data. Junaio allows use of the .MD2 format (a format originating with a video game from over 15 years ago) but throughout my search I was unable to find a .MD2 converter for Blender, Maya 2011, or Cinema 4d. I realized however as I did more research into the other AR platforms function (Layar and Metaio) that I was not able to access animation features at this time. It was at this time that I decided to step back from the 3d element.
Per feedback from class, I decided instead to try and focus on using color tracking and animated 2d imagery in location. It was pointed out to me that it would be nice to replace the abundance of Taxi cabs with something (or anything) else, and these seemed like the perfect target for animated rolling pool balls bouncing down the street. This idea still focused on the mobile idea (desiring to be able to do this within actual New York streets) and therefore still involved mobile platforming. I decided it would be worth it to attempt on the Android platform using Processing for Android.
I wrote a quick color tracking sketch in processing based on Daniel Shiffman’s example from Learning Processing. I also used components from the guicomponent library to allow the user to drag a slider in order to adjust the color on the fly for tracking (with a swatch of the current color to be tracking up in the corner). The sketch looks like this:
(Code is available at the bottom of the page)
As I began to shift this sketch to Android, Zach Schwartz let me know that from a conversation with Daniel Shiffman, the Processing framework is at this time unable to use it’s video libraries with any success on the Android platform. Having realized that this severely limited the projects that Zach and I wanted to do independently, we decided to combine forces and come up with a new project.
We started with Zach’s previous Video Sculpture project where the user revealed pieces of one scene from the horror movie “The Shining.” We had originally planned to continue with the theme of horror and fright.
In discussions about the project, as well as the beginnings of building our code, we quickly realized that the horror circumstance was not going to turn out as we planned. Zach and I left the floor to buy a flashlight to use for our project, and in returning we had come up with a new project. We decided that it was best to embrace the hokey and go with the theme of “finding love” around spots in the city. For the first iteration we decided to go with Washington Square Park to film various scenes around the park, catching park-goers as they continued their daily lives.
Our project included both an infrared camera and a flashlight with a visible light filter (aka developed blank film stolen from the photography department upstairs). As the user pointed the flashlight across a white screen, our projector revealed footage of Washington Square Park, as well as an animated heart that ran from the user (and shrank as it approached the top of the screen). Our video is here: Finding Love in Washington Square Park. Source Code is here.
Feedback from our final class was helpful. While the implementation of the actual hardware was very successful, our content left something to be desired. In the future, I believe that Zach and I would like to return to the horror environment, perhaps revealing things within a diorama setting or chasing shadows just out of reach around a room. All in all, computational cameras taught me a lot about the difficulties and advances in Computer Vision (hint: the Kinect solves a lot of problems out of the box…but not all) and in the future will take the knowledge I have gained into other projects.
Zach also documented this project here.
Taxi Cab Color Tracking:
Thanks to Daniel Shiffman for his Color Tracking example
import guicomponents.*;
import processing.video.*;
Capture video;
color trackColor;
PImage eightball;
int r, g, b;
int thresholdin;
GVertSlider slider;
void setup(){
size(320, 240);
video = new Capture(this, width, height, 30);
thresholdin = 60;
r = 230;
g = 209;
b = 70;
slider = new GVertSlider(this, 0, 0, 35, 100);
eightball = loadImage("8ball.gif");
}
void draw(){
if (video.available()){
video.read();
}
trackColor = color (r, g, b);
video.loadPixels();
image(video, 0, 0);
float worldRecord = 500;
int closestX = 0;
int closestY = 0;
for (int x = 0; x < video.width; x++){
for (int y = 0; y < video.height; y++){
int loc = x + y*video.width;
color currentColor = video.pixels[loc];
float r1 = red (currentColor);
float g1 = green(currentColor);
float b1 = blue(currentColor);
float d = dist(r1, g1, b1, r, g, b);
if (d < worldRecord){
worldRecord = d;
closestX = x;
closestY = y;
}
}
}
fill(trackColor);
rect(width - 40, 0, 40, 40);
noFill();
if (worldRecord < thresholdin){
fill(trackColor);
strokeWeight(4.0);
stroke(0);
image(eightball, closestX, closestY);
//ellipse(closestX, closestY, 16, 16);
}
}
void handleSliderEvents(GSlider slider) {
int sliderV = slider.getValue();
r = int(map(sliderV, 0, 100, 220, 255));
g = int(map(sliderV, 0, 100, 200, 255));
b = int (map(sliderV, 0, 100, 70, 0));
println(r + " " + g + " " + b);
}