Vision

Parsable Space

Telling stories in moving pictures is the most powerful technology ever developed. Unfortunately in the usual progression toward an even more powerful medium that can be parsed and then recombined liked text, motion pictures are stuck in the age of scribes.  Film and video have been kept from having their “movable type moment” because scenes never get broken down beyond the level of the frame.  Each frame is merely a collection of colored pixels instead of a collection of separable and recombinable elements.  We are seeing hint of progress here with elements being captured and manipulated separately in  video games and special effects.  This week are going to take some steps to find some structure in video images.

Getting an Image

From a URL or a  File.

Many of you have already done this.  If you use use a filename rather than a url, you first have to put the file in a “data” folder in your sketch folder.   You can use the “Sketch>Add File” menu to do this in Processing.  Make sure you declare the variable at the top so you can load it in setup but use it in draw.  If you load it over and over in draw() your program will by slow.  The image function can take two coordinates for the position of the image but optionally you can supply two more scaling the width and heigh of the image.

PImage myImage; 

void setup() {
  size(800,600);
  myImage = loadImage("http://fun-media.pl/ftp/photos/unit2/photos/2.10/4/evas_pencils_are_colourful.jpg");
}

void draw( ){
  image(myImage,0,0);
}

 

From the Camera

We are going to make a “Capture” object for pulling in the images from your webcam.  This process is going to feel a little bit like setting up the serial port to listen to Arduino.  Firstly because the camera hardware is different on every platform (eg mac vs pc), getting a hold of them is handled by libraries specific to those platforms.  You have to import that library in the first line (or you could have the “Sketch>Import Library” menu do it for you).  Luckily the correct Capture library for your platform is downloaded with processing.

Next we declare a variable named “video” of type “Capture.”  This will behave just like the PImage object in the earlier example.  You should declare it at the top so it can be created in setup() and used in draw().

In setup() you should create the capture object and put in a variable.  By default it will probably use the web cam on your laptop.  If you want to use another camera attached to your laptop check how to specify that in the documentation of the capture object.  That will feel like picking the port in the Arduino example.  Don’t forget to kickstart the capture object by calling the start function in it (this is newish to Processing so you will see some examples without this).

Finally you want to show the images in draw.  You check to see if a new image is “available” and if so then “read” it. (Note this is similar to the polling method that I discouraged last week for checking for serial bytes coming from Arduino.  Indeed you can use a callback with a captureEvent function like the old serialEvent but I don’t find that it works well).   You use the image() function just as if it were a still image.  From this point on video is just treated the same as a still image.

import processing.video.*;
Capture video;

void setup() {
  size(320,240);
  video = new Capture(this,width,height);
  video.start(); // you might not see this in older examples
}

void draw() {
  // Capture and display the video
  if (video.available()) {
    video.read();
  }
  image(video,0,0);
}

From a Movie

You can also bring movies into processing.  This is a little less interesting to me than the camera because it is not interactive.  You can see this example uses the callback instead of polling style of seeing when new frames came in.  But like with the live video capture, at the end of the day everything is just an image.  You can look at the reference for more options.

import processing.video.*;
Movie myMovie;

void setup() {
  size(200, 200);
  myMovie = new Movie(this, "totoro.mov");
  myMovie.loop();
}

void draw() {
  tint(255, 20);
  image(myMovie, mouseX, mouseY);
}

// Called every time a new frame is available to read
void movieEvent(Movie m) {
  m.read();
}

Getting the Pixels Array from an Image

Images are not very computationally interesting.  They are really take it or leave, all or nothing entities with very little opportunity for interactivity.  We want to break down beyond the frame into the pixels.  Regardless of where your images came from, the internet, a file, a camera, a movie, in the draw loop they are all just images.

Behind every image there is an array of numbers with one integer (int) for each pixel of  the picture.  It is very easy to use dot notation to get at the array of pixels from inside the image object.  We will use this notation to ask about a pixel color or to set a pixel color.

video.pixels[0] // the first pixel of the image called video
video.pixels[55]  //the 56th pixel of the image called video

One Number Per Pixel?  That’s Weird

You could describe grayscale pixels with one number but for color pixels we typically use three separate numbers for red, green and blue.  Luckily an “int” variable is big enough that it can hold four bytes of information, one byte for red, one for green, one for blue and one for alpha or transparency.  This is usually a bummer that all the components of the colors get packed into one number but processing comes to the rescue with some function for packing and unpacking.

int r = red(video.pixels[55]);  //unpacks the red component of the 55th pixel
int g = green(video.pixels[55]); //unpacks the green component of the 55th pixel
int b = blue(video.pixels[55]; //unpacks the blue component of the 55th pixel
int b = brightness (video.pixels[55]; //unpacks the brightness of the 55th pixel
int h = hue(video.pixels[55]; //unpacks the hue of the 55th pixel
int s = saturation(video.pixels[55]; //unpacks the saturation of the 55th pixel

video.pixels[55] = color(255,0,255);  //pack a color with a lot of red and a lot of blue into the 55th pixel

Load Pixels and Update Pixels

Before you start operating on the pixels of an image object you should use the  loadPixels() function of that object to make sure the pixels are fresh.  On the other side once you are done changing the individual pixels of an image object you should use updatePixels() to make sure that array refreshes the image when when it is displayed.  You can get away without doing this quite a lot.

Visit All the Pixels

There are a lot of pixels so you will need a repeat to visit each pixel.   This is a pretty power thing to be able to visit hundreds of thousands of pixels every 30 milliseconds (30 frames/sec) and ask each one a question (how bright) and set it to a new color accordingly.

import processing.video.*;

Capture video; 

void setup() {
  size(320, 480);
  video = new Capture(this, 320, 240);
  video.start();
}

void draw( ) {
  if ( video.available() ) {
    video.read();  //read in the new image if it is available
  }
  for (int i = 0; i < video.pixels.length;  i++) {  //you can ask the pixel array how long it is
    if ( brightness(video.pixels[i]) > 127) {  //check the brightness of the pixel
      video.pixels[i] = color(255, 0, 255);  //set the pixel to purble if it is bright
    }
  }
  image(video, 0, video.height);
}

Try changing this code to look for edges by comparing each pixel to the previous.

Image Processing vs Computer Vision

As powerful as that repeat loop is, it is not good enough.  The code above does image processing and then the results are delivered to your eyeballs to interpret.  We want the machine to do a little more interpretation itself, for instance separate an object from the background and tell me the position of the object in the scene.  Rather than delivering the results only to your eyeballs I  want it to deliver the results in numbers that can then be used in your code for other things.  For instance you could have an if statement that says if their x position is greater than 100 pour a virtual bucket of water on them. Or play a higher pitched not as your hand moves up.   Your next goal to should be to track the position of something in the camera’s view.

Art School vs Homeland Security

You should contrive your the physical side of your tracking situation as much as possible to make the software side as easy as possible.  Computer Vision is pretty hard if you are trying to find terrorists at the super bowl.  But your users have a stake in the the thing working and will play along.  Rather than tracking natural things it is okay to have the user wear a big orange oven mitt.

The Classic: Rows and Columns and Questions

Unfortunately the pixels are given to you in one long array of numbers but we think of images in terms of rows and columns.  Instead of a single for loop we will have a “for loop” inside  a “for loop.”  The first loop will visit every column and the second will visit each row in the column (same to do the reverse visit each row and than each column in the row).  Then you pull out the colors of the pixel and ask a question about them (more on the possible questions below).  This “for loop” within a “for loop” , extracting color components followed by an if statement question will be the heart of every computer vision program you ever write.

 for (int x = 0; x < video.width; x ++ ) {
    for (int y = 0; y < video.height; y ++ ) {
      int loc = x + y*video.width;
      // What is current color
      color currentColor = video.pixels[loc];
      int red = red(currentColor);
      int green = green(currentColor);
      int blue = blue(currentColor);
      SOME IF STATEMENT THAT ASKS A QUESTION ABOUT THAT COLOR
     }
 }

 Formula for Location in Pixel Array for a Given Row and Column

There is just one trick in there for finding the offset in the long linear array of numbers for a given row and column location.  The formula is:

int loc = x + y*video.width;

You perform this formula every time you try to figure out how many people are in a theater.  Pretending people filled all the seats in order you would find the number seats in a row and multiply times how many rows are full and then add how many seats are in the last partially full row.

Comparing Colors

As we will see, after you extract the colors of the pixel you might want to compare it to a color you are looking for, the color of the adjacent pixel, the same pixel in the last frame etc….  You could just subtract and use absolute value to find the difference in color.  But it turns out in color space as in real space finding the distance as the crow flies between two colors is better done with the pythagorean theorem.  Coming to the rescue in Processing is the dist() function.

   float r1 = red(currentColor);
   float g1 = green(currentColor);
   float b1 = blue(currentColor);
   float r2 = red(trackColor);
   float g2 = green(trackColor);
   float b2 = blue(trackColor);

   // Using euclidean distance to compare colors
   float d = dist(r1,g1,b1,r2,g2,b2); 
   // We are using the dist( ) function to compare the current color with the color we are tracking.

Things You Can Ask Pixels

Color Tracking: Which Pixel Is Closest to A Particular Color?

In this code we will use three variables, one to keep track of how close the best match yet is.  And two more to store the location of the best match so far.

You click on a pixel to set the color you are tracking.  Notice we used this formula again in mousePressed() find the offset in the big long array for the particular row and column you clicked at.  Remember it is usually better to insert some patch of unnatural color, say a bright colored toy, into your scene than to make software that  can be very discerning of nature.

void mousePressed() {
  // Save color where the mouse is clicked in trackColor variable
  int loc = mouseX + mouseY*video.width;
  trackColor = video.pixels[loc];
}
// Learning Processing
// Daniel Shiffman
// http://www.learningprocessing.com

// Example 16-11: Simple color tracking

import processing.video.*;

// Variable for capture device
Capture video;

// A variable for the color we are searching for.
color trackColor; 

void setup() {
  size(320, 240);
  video = new Capture(this, width, height);
  video.start();
  // Start off tracking for red
  trackColor = color(255, 0, 0);
}

void draw() {
  // Capture and display the video
  if (video.available()) {
    video.read();
  }
  video.loadPixels();
  image(video, 0, 0);

  // Before we begin searching, the "world record" for closest color is set to a high number that is easy for the first pixel to beat.
  float worldRecord = 500; 

  // XY coordinate of closest color
  int closestX = 0;
  int closestY = 0;

  // Begin loop to walk through every pixel
  for (int x = 0; x < video.width; x ++ ) {
    for (int y = 0; y < video.height; y ++ ) {
      int loc = x + y*video.width;
      // What is current color
      color currentColor = video.pixels[loc];
      float r1 = red(currentColor);
      float g1 = green(currentColor);
      float b1 = blue(currentColor);
      float r2 = red(trackColor);
      float g2 = green(trackColor);
      float b2 = blue(trackColor);

      // Using euclidean distance to compare colors
      float d = dist(r1, g1, b1, r2, g2, b2); // We are using the dist( ) function to compare the current color with the color we are tracking.

      // If current color is more similar to tracked color than
      // closest color, save current location and current difference
      if (d < worldRecord) {
        worldRecord = d;
        closestX = x;
        closestY = y;
      }
    }
  }

  // We only consider the color found if its color distance is less than 10. 
  // This threshold of 10 is arbitrary and you can adjust this number depending on how accurate you require the tracking to be.
  if (worldRecord < 10) { 
    // Draw a circle at the tracked pixel
    fill(trackColor);
    strokeWeight(4.0);
    stroke(0);
    ellipse(closestX, closestY, 16, 16);
  }
}

void mousePressed() {
  // Save color where the mouse is clicked in trackColor variable
  int loc = mouseX + mouseY*video.width;
  trackColor = video.pixels[loc];
}

 

Smoother Color Tracking: What is the Average Location of of Pixels That are Close Enough Particular Color?

The key part of this code is the fact that I am not looking for a single pixel but instead using a threshold to find a group of pixels that are close enough.  In this case I will average the locations of all the pixels that were close enough.  In the previous example a bunch of similar pixels would win the contest of being closest in different frames so it was kind of jumpy.  Now the whole group of close ones win so it will be smoother.

   if (diff < threshold) {  //if it is close enough in size, add it to the average
          sumX = sumX + col;
          sumY= sumY + row;
          totalFoundPixels++;
          if (debug) cam.pixels[offset] = 0xff000000;//debugging
        }

Another thing to notice in this code is the key pressed function.  That is used to change variable, especially the threshold variable dynamically while the program is running.  This is important in computer vision because light conditions can change so easily.

void keyPressed() {
  //for adjusting things on the fly
  if (key == '-') {
    threshold--;
    println("Threshold " + threshold);
  } 
  else if (key == '=') {
    threshold++;
    println("Threshold " + threshold);
  }
  else if (key == 'd') {
    background(255);
    debug = !debug;
    println("Debug " + debug);
  }
  else if (key == 't') {
    println("Time Between Frames " + ellapsedTime);
  }
}

This is not very important to don’t read it but I using a timer to test the performance.  As fast as computers are these days, you will really be testing them with computer vision where you need to visit millions of pixels per second.  If you add a third repeat loop in there you will really need to start checking performance.  We have seen this method before of using a variable to store the millis() and then checking the millis() against that variable to see how much time has passed.  There are about 33 millisecond in a frame of video at 30 frames/second.

ellapsedTime = millis() - lastTime;  
//find time since last time, only print it out if you press "t"
lastTime = millis();  //reset timer for checking time next frame

Okay here is some smoother tracking.

//tracking the average pixel makes it shake less
import processing.video.Capture;
Capture cam;// regular processing libary
int threshold = 20; //255 is white, 0 is black
int aveX, aveY; //this is what we are trying to find
float objectR =255;
float objectG = 0;
float objectB = 0;
boolean debug = true;
int lastTime, ellapsedTime; //for checking performance
void setup() {
  size(640, 480);
  println(Capture.list());
  cam = new Capture(this,width,height);
  cam.start();
}
void draw(){
  if (cam.available()){
    ellapsedTime = millis() - lastTime;  //find time since last time, only print it out if you press "t"
    lastTime = millis();  //reset timer for checking time next fram
    cam.read();
    int totalFoundPixels= 0;  //we are going to find the average location of change pixes so
    int sumX = 0;  //we will need the sum of all the x find, the sum of all the y find and the total finds
    int sumY = 0;
    //enter into the classic nested for statements of computer vision
    for (int row = 0; row < cam.height; row++) {
      for (int col = 0; col < cam.width; col++) {
        //the pixels file into the room long line you use this simple formula to find what row and column the sit in 

        int offset = row * cam.width + col;
        //pull out the same pixel from the current frame 
        int thisColor = cam.pixels[offset];

        //pull out the individual colors for both pixels
        float r = red(thisColor);
        float g = green(thisColor);
        float b = blue(thisColor);

        //in a color "space" you find the distance between color the same whay you would in a cartesian space, phythag or dist in processing
        float diff = dist(r, g, b, objectR, objectG, objectB);

        if (diff < threshold) {  //if it is close enough in size, add it to the average
          sumX = sumX + col;
          sumY= sumY + row;
          totalFoundPixels++;
          if (debug) cam.pixels[offset] = 0xff000000;//debugging
        }
      }
    }
    image(cam,0,0);
    if (totalFoundPixels > 0){
      aveX = sumX/totalFoundPixels;
      aveY = sumY/totalFoundPixels;
      ellipse(aveX-10,(aveY-10),20,20);
    }
  }
}
void mousePressed(){
  //if they click, use that picture for the new thing to follow
  int offset = mouseY * cam.width + mouseX;
  //pull out the same pixel from the current frame 
  int thisColor = cam.pixels[offset];

  //pull out the individual colors for both pixels
   objectR = red(thisColor);
   objectG = green(thisColor);
   objectB = blue(thisColor);
  println("Chasing new color  " + objectR + " " + objectG + " " + objectB);
}
void keyPressed() {
  //for adjusting things on the fly
  if (key == '-') {
    threshold--;
    println("Threshold " + threshold);
  } 
  else if (key == '=') {
    threshold++;
    println("Threshold " + threshold);
  }
  else if (key == 'd') {
    background(255);
    debug = !debug;
    println("Debug " + debug);
  }
  else if (key == 't') {
    println("Time Between Frames " + ellapsedTime);
  }
}

 

Looking for Change: How does this Pixel Compare to the background?

For this we need to store another set of background pixels to compare the incoming frame to.  If you use the previous frame as the background pixels you will be tracking change.  If you set the background in setup or in mousepressed() (as in this example) you will be doing background removal.

// Learning Processing
// Daniel Shiffman
// http://www.learningprocessing.com

// Example 16-12: Simple background removal

// Click the mouse to memorize a current background image
import processing.video.*;

// Variable for capture device
Capture video;

// Saved background
PImage backgroundImage;

// How different must a pixel be to be a foreground pixel
float threshold = 20;

void setup() {
  size(320,240);
  video = new Capture(this, width, height, 30);
  video.start();
  // Create an empty image the same size as the video
  backgroundImage = createImage(video.width,video.height,RGB);
}

void draw() {
  // Capture video
  if (video.available()) {
    video.read();
  }

  // We are looking at the video's pixels, the memorized backgroundImage's pixels, as well as accessing the display pixels. 
  // So we must loadPixels() for all!
  loadPixels();
  video.loadPixels(); 
  backgroundImage.loadPixels();

  // Begin loop to walk through every pixel
  for (int x = 0; x < video.width; x ++ ) {
    for (int y = 0; y < video.height; y ++ ) {
      int loc = x + y*video.width; // Step 1, what is the 1D pixel location
      color fgColor = video.pixels[loc]; // Step 2, what is the foreground color

      // Step 3, what is the background color
      color bgColor = backgroundImage.pixels[loc];

      // Step 4, compare the foreground and background color
      float r1 = red(fgColor);
      float g1 = green(fgColor);
      float b1 = blue(fgColor);
      float r2 = red(bgColor);
      float g2 = green(bgColor);
      float b2 = blue(bgColor);
      float diff = dist(r1,g1,b1,r2,g2,b2);

      // Step 5, Is the foreground color different from the background color
      if (diff > threshold) {
        // If so, display the foreground color
        pixels[loc] = fgColor;
      } else {
        // If not, display green
        pixels[loc] = color(0,255,0); // We could choose to replace the background pixels with something other than the color green!
      }
    }
  }
  updatePixels();
}

void mousePressed() {
  // Copying the current frame of video into the backgroundImage object
  // Note copy takes 5 arguments:
  // The source image
  // x,y,width, and height of region to be copied from the source
  // x,y,width, and height of copy destination
  backgroundImage.copy(video,0,0,video.width,video.height,0,0,video.width,video.height);
  backgroundImage.updatePixels();
}

 

Skin Rectangles: Is this pixel skin colored and is it part of an existing group of pixels?

Okay this example is combining two new things.  The most important thing is we are looking for multiple groups of pixels so find a single average for the whole frame as we were doing won’t be enough.  Instead we are going to ask each pixel first if they qualify against a threshold and then second if they are close to an existing group or whether they need to start a new group.  This could be used for pixels that qualify for things other than  skin colore, for example the pixels could qualify for being a bright or being part of the foreground.

    //is this spot in an  existing box
          if (existingBox.isNear(col, row,reach)) {
            existingBox.add(col, row);
            foundAHome = true; //no need to make a new one
            break; //no need to look through the rest of the boxes
          }
        }
        //if this does not belong to one of the existing boxes make a new one at this place
        if (foundAHome == false) boxes.add(new Rectangle(col, row));
      }

This example also happens to be about skin color.  It is a heart warming fact that regardless of race we are all pretty much the same color because we all have the same blood.  We are however different brightnesses.  Because brightness is not a separable notion in the RGB color space we using a “normalized” color space where we are looking for percentages of red and green (the third color will just be the remain percentage).

  float r = red(_thisPixel);
  float g = green(_thisPixel);
  float total = r + g + blue(_thisPixel);
//convert into a "normalized" instead of RGB
  float percentRed = r/total;
  float percentGreen = g/total;
  return (percentRed < skinRedUpper && percentRed > skinRedLower  && percentGreen < skinGreenUpper && percentGreen > skinGreenLower);

Okay there is the code in two parts.  There is also a rectangle object for holding the the qualifying pixels.

import processing.video.*;

float skinRedLower = .35f;
float skinRedUpper = .55f;
float skinGreenLower = .26f;
float skinGreenUpper = .35f;
int reach = 3;
Capture cam;
int w = 640;
int h = 480;
boolean debug = true;
long elapsedTime = 0;

void setup() {
  size(w, h);  
  cam = new Capture(this, w, h, 30);
  cam.start();
}

public void draw() {  //called everytime there is a new frame

  long startTime = System.currentTimeMillis();

  if (cam.available()) {
    cam.read(); //get the incoming frame as a picture
    ArrayList rects = findRectangles();

    //println(elapsedTime);  
    consolidate(rects, 0, 0);
    cleanUp(rects, 100);
    if (debug) image(cam, 0, 0);
    fill(0, 0, 0, 0);
    stroke(255, 0, 0);
    for (int i = 0; i < rects .size(); i++) {
      Rectangle thisBox =  (Rectangle) rects .get(i);
      thisBox.draw();
    }

    elapsedTime = System.currentTimeMillis() - startTime;
  }
}

boolean test(int _thisPixel) {
  float r = red(_thisPixel);
  float g = green(_thisPixel);
  float total = r + g + blue(_thisPixel);
//convert into a "normalized" instead of RGB
  float percentRed = r/total;
  float percentGreen = g/total;
  return (percentRed < skinRedUpper && percentRed > skinRedLower  && percentGreen < skinGreenUpper && percentGreen > skinGreenLower);
}

void keyPressed() {

  //for adjusting things on the fly
  if (key == '-') {
    skinRedUpper = skinRedUpper - .01f;

    skinRedLower = skinRedLower - .01f;
  } 
  else if (key == '=') {
    skinRedUpper = skinRedUpper + .01f;

    skinRedLower = skinRedLower + .01f;
  }
  else if (key == '_') {
    skinGreenUpper = skinGreenUpper - .01f;

    skinGreenLower = skinGreenLower - .01f;
  } 
  else if (key == '+') {
    skinGreenUpper = skinGreenUpper + .01f;

    skinGreenLower = skinGreenLower + .01f;
  }
  else if (key == 'r') {
    reach--;
    println("reach " + reach);
  } 
  else if (key == 'R') {
    reach++;
    println("reach " + reach);
  } 
  else if (key == 't') {

    println("Elapsedtime " + elapsedTime);
  }
  else if (key == 'd') {
    debug = !debug;
    println("debug  " + debug);
  }
  println("RU:" + skinRedUpper + " RL:" + skinRedLower +"GU:" + skinGreenUpper + " GL:" + skinGreenLower );
}
void cleanUp(ArrayList _rects, int _sizeThreshold) {
  for (int j = _rects.size() - 1; j > -1; j--) { 

    Rectangle newRect = (Rectangle) _rects.get(j);
    if (newRect.getHeight()*newRect.getWidth() < _sizeThreshold) _rects.remove(j); //if the area to small, loose it
  }
} 

public void consolidate(ArrayList _shapes, int _consolidateReachX, int _consolidateReachY) { 

  //check every combination of shapes for overlap 
  //make the repeat loop backwards so you delete off the bottom of the stack
  for (int i = _shapes.size() - 1; i > -1; i--) {
    //only check the ones up 
    Rectangle shape1 = (Rectangle) _shapes.get(i);

    for (int j = i - 1; j > -1; j--) {
      Rectangle shape2 = (Rectangle) _shapes.get(j);
      if (shape1.intersects(shape2) ) {
        shape1.add(shape2);
        //System.out.println("Remove" + j);
        _shapes.remove(j);
        break;
      }
    }
  }
} 
ArrayList findRectangles() {
  ArrayList boxes = new ArrayList();

  for (int row = 0; row < cam.height; row++) {
    for (int col = 0; col < cam.width; col++) {
      int offset = row * width + col;
      int thisPixel = cam.pixels[offset];
      if (test(thisPixel)) {
        cam.pixels[offset] = 0xffff00;
        //be pessimistic
        boolean foundAHome = false;
        //look throught the existing
        for (int i = 0; i < boxes.size(); i++) {
          Rectangle existingBox =  (Rectangle) boxes.get(i);

          //is this spot in an  existing box
          if (existingBox.isNear(col, row,reach)) {
            existingBox.add(col, row);
            foundAHome = true; //no need to make a new one
            break; //no need to look through the rest of the boxes
          }
        }
        //if this does not belong to one of the existing boxes make a new one at this place
        if (foundAHome == false) boxes.add(new Rectangle(col, row));
      }
    }
  }
  return boxes;
}

 

class Rectangle {
  public int furthestLeft;
  public int furthestUp;
  public int furthestRight;
  public int furthestDown;

  Rectangle(int _x, int _y) {
    furthestLeft = _x;
    furthestUp = _y;
    furthestRight = _x;
    furthestDown = _y;
  }

  void add(int _x, int _y) {
    if (_x < furthestLeft) furthestLeft = _x;
    if (_x > furthestRight) furthestRight = _x;
    if (_y < furthestUp) furthestUp = _y;
    if (_y > furthestDown) furthestDown = _y;
  }

  void add(Rectangle _rect){
    if (_rect.furthestLeft < furthestLeft) furthestLeft = _rect.furthestLeft;
    if (_rect.furthestRight  > furthestRight) furthestRight = _rect.furthestRight ;
    if (_rect.furthestUp   < furthestUp) furthestUp = _rect.furthestUp ;
    if (_rect.furthestDown  > furthestDown) furthestDown = _rect.furthestDown ;
  }

  boolean isNear(int _x, int _y, int _reach) {
    //make sure this new spot is inside the current stretch by reach
   return ((_x >= furthestLeft-_reach && _x < furthestRight + _reach) && (_y >= furthestUp -reach && _y < furthestDown + _reach));
  }

  void draw() {
    rect(furthestLeft, furthestUp, furthestRight-furthestLeft, furthestDown-furthestUp);
  }

  int getWidth(){
    return furthestRight-furthestLeft;
  }

  int getHeight(){
    return furthestDown-furthestUp;
  }

  boolean intersects(Rectangle _other){
    return ! ( _other.furthestLeft > furthestRight
    || _other.furthestRight < furthestLeft
    || _other.furthestUp > furthestDown
    || _other.furthestDown < furthestUp
    );
  }
}

 

Other Examples:

Face detection

Kinect

Network Camera

Related reading:
Learning Processing, Chapters 15-16
Assignment

Pixels Project

Track a color and have some animation or sound change as a result.  Create a software mirror by designing an abstract drawing machine which you color according to pixels from live video.
Create a video player. Consider combining your pcomp media controller assignment and build a Processing sketch that allows you to switch between videos, process pixels of a video, scrub a video, etc.
Use the kinect to track a skeleton. Can you “puppeteer” an avatar/animation with the kinect?

Leave a Reply