Kinect

Images

It is easy to put images in  processing.  There are two commands, loadImage (in setup) and image (in draw) that works pretty much like the rect and ellipse command but draws an image.  This is the same pattern as putting sound into your program.  Load the sound in setup() and use it in draw().

PImage thrust;

void setup() {
 size(500, 500 );
 //load image from a file or a url
 thrust = loadImage("http://media.giphy.com/media/satnVmCFv0YA8/giphy.gif");
}

void draw(){
 image(thrust,0,0);
}

The problem with the image command, is that it is pretty much take it or leave it.  You can’t change or analyze the image.  This week we are going to dive into the pixels of the images coming in from a camera to analyze and change the image.

Images as Arrays

Images are stored as a collection of pixels.  As we learned last week, if your program wants to wants to change those images or analyze the pixlels we are going to have to put them into variables.   You would need a lot of variables to have one for each pixel of even a small picture.  To the rescue there is a special type of variable called an array that has many slots in it.  We can address the individual slots in the array by number (starting with zero).  It is sort of like this pill box named myvar with the number 169 stored in the second slot. You use square brackets to say which compartment you want to address. To put 169 in  you would say myvar[1] = 169.  To pull it out you would say int num = myVar[1]  (I know it is weird that the second slot has an index of 1 but you just have to get used to that);

pillBox

So we store the pixels of an image in a big long array, maybe 500,000 slots.  The pixels for the programs you have made so far were stored for you in a variable that processing makes automatically called pixels.  On a less interesting note, to pull the pixels off the screen into the array you should say loadPixels() and to pull the pixels out of the array back onto the screen you should say updatePixels().  This program sets those pixels.

void setup(){
 size(100,100);
 
}

void draw(){
 loadPixels(); //don't worry about this 
 pixels[400] = color(0,0,0);
 pixels[700] = color(0,0,0);
 pixels[1100] = color(0,0,0);
 //paint a pixel where the mouse is
 pixels[mouseY*width+ mouseX] = color(255,255,0);
 updatePixels();//don't worry about this
}

Arrays and For Loops

Now we don’t need a million variables for a million pixels because we can just have an array variable with a million slots.  Unfortunately we would still need a million lines of code to pull the pixels out and to set them.  Coming to the rescue is the for loop.  A for loop allows you to repeat a code block (curly brackets) while incrementing a variable to keep track of everything.  The syntax is the word for followed by a parenthesis with three things.  Here is the psuedocode.  Notice  the three parts are separated by semicolons.

for (declare/initialize variable ; limit how big ; how to change variable each loop){
//stuff to repeat
}

void setup(){
  size(100,100);
  
}

void draw(){
  loadPixels(); //don't worry about this  
  // variable i starts at 0, goes to 4999, gets bigger by one each time
  for(int i = 0; i < 5050; i++){
     pixels[i] = color(0,0,0);
  }
  updatePixels();//don't worry about this
}

When was the last time you did 5050 things so quickly.  Do you feel the power?  This loop makes an int variable called “i” and puts  zero in it to start.  For some reason programmers love using the variable named i in for loops but you can name it anything.   It will keep doing  the code in the code block over and over until the gets to 5050.  Each time through the variable will get bigger by 1.  i++ is the same thing as saying i = i + 1;

For a video with your favorite explainer check this out.

Going through the pixels coming in from a live camera is the same as looking though the pixels of a still image except you do it 30 time a second.  So the draw loop happens 30 times a second for each frame of video and then a nested for loop happens 320,000 times for each draw loop, once for each pixel.

Live Camera

import processing.video.*;
Capture video;

void setup() {
  size(320,240);
  video = new Capture(this,width,height);
  video.start(); // you might not see this in older examples
}

void draw() {
  // Capture and display the video
  if (video.available()) {
    video.read();
  }
  image(video,0,0);
}

Look at Pixels in Live Camera

import processing.video.*;

Capture video; 

void setup() {
  size(320, 480);
  video = new Capture(this, 320, 240);
  video.start();
}

void draw( ) {
  if ( video.available() ) {
    video.read();  //read in the new image if it is available
  }
  for (int i = 0; i < video.pixels.length;  i++) {  //you can ask the pixel array how long it is
    if ( brightness(video.pixels[i]) > 127) {  //check the brightness of the pixel
      video.pixels[i] = color(255, 0, 255);  //set the pixel to purble if it is bright
    }
  }
  image(video, 0, video.height);
}

 

Enter the Kinect

Pixels tell you what color you are but in giving expression to your body you might care more about where your body is rather than what color it is.  The kinect gives you not only an array with colors for each pixel but gives you another array with the depth of each pixel.  Go ahead and install the Kinect.

Install The Kinect Library

  1. In Processing find the menus Sketch>Import Library> Add Library to get a window of available libraries.
  2. Scroll down to “Open Kinect for Processing by Dan Shiffman and Thomas Sanchez” and install it, and close the window.  If you are on a PC you need to run  a program recommended here to get your machine ready.  This is a different and fancier processing library for the PC  that requires that you install something (SDK) from Microsoft.
  3. I usually restart Processing but I think that might be superstition.
  4. Try going the menu Files > Examples… to see the fun stuff you can do.
  5. Import the library into your program by putting these lines at the very top import org.openkinect.freenect.*;  and import org.openkinect.processing.*;   The menu Sketch>Import Library> Open Processing for Kinect will do this for you.
import org.openkinect.freenect.*;
import org.openkinect.processing.*;

Using the Kinect Library

  1. Make a variable at the top to hold your Kinect object.  Instead of and integer or floating point, it should be of type Kinect.
  2. In setup you have to put a new Kinect object into that variable and then enable a few parts of the Kinect.
  3. In the draw loop go looking through all the the depths of the pixels coming in from the Kinect.
  4. Do something based on some change in depth of the pixels.
  5. Here is the documentation for this library for more info.

Simple Pelvic Thrust

import org.openkinect.freenect.*;
import org.openkinect.processing.*;

Kinect kinect;

void setup() {
  size(500, 500 );
  kinect = new Kinect(this);
  kinect.initDepth();
  kinect.initVideo();
  kinect.enableMirror(true);
}

void draw() {
  //if the kinect has not warmed up yet bail, bug?
  if (kinect == null) return;

  //paint the color image on the screen
  image(kinect.getVideoImage(), 0, 0, width, height);
  //ask kinect for an array of depth pixels
  int[] depth = kinect.getRawDepth();

  //look through all the pixels
  for (int i = 0; i < depth.length; i++) {
    int rawDepth = depth[i];
    //check if it falls between these depths
    if (rawDepth > 500 && rawDepth < 600) {
      //draw a red rectangle
      stroke(0, 255, 0);
      fill(255, 0, 0);
      rect(100, 100, width-200, height-200);
      break; //don't bother checking the rest, cancel loop
    }
  }
}

Here is one a little fancier where it has a smaller target within the screen.  For this it need to scan through the pixels in the long array by row and column.  There are variables for the borders of the target within the screen.

Bug Fix in Libary for Kinect model 1473
//this tries to start up the kinect when you press any key
//you need this for model 1473 model kinect because of bug in library
//press a key if the kinect initially doesn't work
void keyPressed(){
    kinect = new Kinect(this);
    kinect.initDepth();
    kinect.initVideo();
    kinect.enableMirror(true);
}

 

Vary Position of the Target

import org.openkinect.freenect.*;
import org.openkinect.processing.*;
Kinect kinect;

int top = 120;
int left = 100;
int zoneSize = 100;


void setup() {
  size(500, 500 );
  kinect = new Kinect(this);
  kinect.initDepth();
  kinect.initVideo();
  kinect.enableMirror(true);
}

void draw() {
  //if the kinect has not warmed up yet bail, bug?
  if (kinect == null) return;
  
  //paint the color image on the screen
  image(kinect.getVideoImage(), 0,0, width, height);
  //ask kinect for an array of depth pixels
  int[] depth = kinect.getRawDepth();

  //draw rect
  stroke(0, 0, 255);
  fill(0, 0, 0, 0);
  rect(left,top, zoneSize, zoneSize);

  //go is for loop through all the pixels
  for (int x = left; x < left + zoneSize; x++) {
    for (int y = top; y < top + zoneSize; y++) {   
      //find the position of this pixel in the long array
      int offset =  x + y*kinect.width;
      // find the depth of this pixel
      int rawDepth = depth[offset];
      //check if it falls 
      if (rawDepth > 0 && rawDepth < 600) {
        //draw a red rectangle
        stroke(0, 255, 0);
        fill(255,0,0);
        rect(left,top, zoneSize, zoneSize);
        break; //don't bother testing the rest
      }
    }//end of for y
  }//end for x
}

Add this little function at the bottom to allow you to change the variables with the keyboard while the program is running.

//convenient function for moving around the area that you care about
void keyPressed(){
  if(key == 't'){
    top = top -10;
  }else if(key == 'T'){
    top = top + 10;
  }
   if(key == 'l'){
    left = left - 10;
  }else if(key == 'L'){
    left = left + 10;
  }
}

Overlay Another Image of Depth Pixels

You can make a blank image at the beginning, change the pixels of that depending on the depth and then overlay that on the full color video image you painted at the top of the draw loop.  These are the lines added to make a new image and overlay it:

//make a new image before for loop
PImage depthImg = new PImage(kinect.width, kinect.height, ARGB);


//set pixels in the new image in the for loop
depthImg.pixels[offset] = color(dColor, dColor, dColor, 127);


//after the for loop update the pixels and paint them on screen
depthImg.updatePixels();
image(depthImg, 0, 0);

Here they are integrated into the previous example:

import org.openkinect.freenect.*;
import org.openkinect.processing.*;
Kinect kinect;

int top = 120;
int left = 100;
int zoneSize = 100;


void setup() {
  size(500, 500 );
  kinect = new Kinect(this);
  kinect.initDepth();
  kinect.initVideo();
  kinect.enableMirror(true);
}

void draw() {
  //if the kinect has not warmed up yet bail, bug?
  if (kinect == null) return;

  //paint the color image on the screen
  image(kinect.getVideoImage(), 0, 0, width, height);
  //ask kinect for an array of depth pixels
  int[] depth = kinect.getRawDepth();

  //draw rect
  stroke(0, 0, 255);
  fill(0, 0, 0, 0);
  rect(left, top, zoneSize, zoneSize);

  PImage depthImg = new PImage(kinect.width, kinect.height, ARGB);

  //go is for loop through all the pixels
  for (int x = left; x < left + zoneSize; x++) {
    for (int y = top; y < top + zoneSize; y++) {   
      //find the position of this pixel in the long array
      int offset =  x + y*kinect.width;
      // find the depth of this pixel
      int rawDepth = depth[offset];
      //check if it falls 
      if (rawDepth > 0 && rawDepth < 600) {
        //draw a red rectangle
        stroke(255, 0, 0);
        fill(0, 0, 0, 0);//transparent
        rect(left, top, zoneSize, zoneSize);
        //map the depth numbers to color numbers
        float dColor = map(rawDepth, 0, 600, 0, 255);
        //all the same number makes gray, 127 is half transparent
        depthImg.pixels[offset] = color(dColor, dColor, dColor, 127);
      }
    }//end of for y
  }//end for x
  //pull the pixels out of the array and paint them the screen
  depthImg.updatePixels();
  image(depthImg, 0, 0);
}

More Than On Off

This is even a little fancier.  It does more than just find whether or not there is a single pixel within that target, it find the average depth within the target and uses that play a sound file a different volumes.

import org.openkinect.freenect.*;
import org.openkinect.processing.*;

Kinect kinect;
int[] depth;
PImage depthImg;
PImage colorImg;

int top = 50;
int left = 50;
int zoneSize = 200;


void setup() {
  size(640, 520);
  kinect = new Kinect(this);
  kinect.initDepth();
  kinect.initVideo();
  kinect.enableMirror(true);
  depthImg = new PImage(kinect.width, kinect.height, ARGB);
}

void draw() {

  colorImg = kinect.getVideoImage();
  image(kinect.getVideoImage(), 0, 0);

  depth = kinect.getRawDepth();

  //variables for keeping track of average depth
  float sumDepth = 0;
  float count = 0;

  stroke(0,0,255);
  fill(0,0,0,0);
  rect(left,top,zoneSize,zoneSize);
  
  for (int x = left; x < left + zoneSize; x++) {
    for (int y = top; y < top + zoneSize; y++) {       
      int offset =  x + y*kinect.width;
      // Grabbing the raw depth
      int rawDepth = depth[offset];
      if (rawDepth >= 0 && rawDepth <= 600 ) {
        depthImg.pixels[offset] = color(255,0,0,100);
        //add together the depth of all the qualifying pixels to average later.
        sumDepth = sumDepth + rawDepth ;
        count++;
      } else {
        depthImg.pixels[offset] = color(0,0,0,0);
      }
    }
  }
  depthImg.updatePixels();
  image(depthImg, 0, 0);
  if (count > 3){ //make sure there were a significant bunch of pixels
    //average the depths of all the qualifying pixesl
    float aveDepth = sumDepth/count;
    //map the average depth to color numbers
    float analogNumber = map( aveDepth , 0, 600, 0, 255);
    //the tint command tints the whole screen!
    tint(analogNumber,analogNumber,analogNumber,analogNumber);
    println(aveDepth);
  }
}

Multiple Targets (cleaned up by making a function)

And finally you can have multiple targets if you farm your code into a resuable custom function that you name and write.  In this case I wrote a new function called getAverageDepth() that encapsulates all the code in that repeat loop.  If I send it different parameters it can do the same repeat loop for different positions.

import org.openkinect.freenect.*;
import org.openkinect.processing.*;
 
Kinect kinect;
 
void setup() {
  size(500, 500);
  kinect = new Kinect(this);
  kinect.initDepth();
  kinect.initVideo();
  kinect.enableMirror(true);
}
 
void draw() {
  PImage colorBackground = kinect.getVideoImage();
  image(colorBackground, 0, 0);
 
  PImage showDepthImg = new PImage(kinect.width, kinect.height, ARGB);
 
  //make one target
  float val = getAverageDepth(0, 0, 200, 300, 800, showDepthImg);
  fill(0);
  text("Ave Depth:"+ val, 100,100);
 
  //make a second target
  float val2 = getAverageDepth(width-200, 0, 100, 300, 800, showDepthImg);
  fill(0);
  text("Ave Depth:"+ val2, width-150,50);
  
  image(showDepthImg, 0, 0);
}
 
 
 
 
float getAverageDepth(int hotSpotX, int hotSpotY, int hotSpotSize, int frontDepth, int backDepth, PImage debugImg) {
  //variables for keeping track of average depth
  float sumDepth = 0;
  float count = 0;
 
  stroke(0, 0, 255);
  fill(0, 0, 0, 0);
  rect(hotSpotX, hotSpotY, hotSpotSize, hotSpotSize);
 
  int[] depth = kinect.getRawDepth();
  for (int x = hotSpotX; x < hotSpotX + hotSpotSize; x++) {
    for (int y = hotSpotY; y < hotSpotY + hotSpotSize; y++) {       
      int offset =  x + y*kinect.width;
      // Grabbing the raw depth
      int rawDepth = depth[offset];
      if (rawDepth >= frontDepth && rawDepth <= backDepth ) {
        if (debugImg !=null) debugImg.pixels[offset] = color(255, 0, 0, 100);
        //add together the depth of all the qualifying pixels to average later.
        sumDepth = sumDepth + rawDepth ;
        count++;
      }
    }
  }
  float averageDepth = 0.0;
  if (count > 3) { //make sure there were a significant bunch of pixels
    //average the depths of all the qualifying pixesl
    averageDepth = sumDepth/count;
    //map the average depth to color numbers
    averageDepth = map( averageDepth, frontDepth, backDepth, 0, 255);
  }
  if (debugImg !=null) debugImg.updatePixels();
  return averageDepth;  //send back average depth of things in this area.
}

//you need this for model 1473 model kinect because of bug in library
void keyPressed(){
    kinect = new Kinect(this);
    kinect.initDepth();
    kinect.initVideo();
    kinect.enableMirror(true);
}

Find Depth Blogs

Added later after class

import org.openkinect.freenect.*;
import org.openkinect.processing.*;

Kinect kinect;
int reach = 3;
int reachZMultiplier = 4;
int minSize = 2000;

void setup() {
  size(1280, 480);
  kinect = new Kinect(this);
  kinect.initDepth();
  kinect.initVideo();
  kinect.enableMirror(true);
  noFill();
  rectMode(CORNERS);
}

void draw() {
 // PImage colorBackground = kinect.getVideoImage();
  //image(colorBackground, 0, 0);
  background(255);
  PImage showDepthImg = new PImage(kinect.width, kinect.height, ARGB);
  image(kinect.getDepthImage(), 0, 0);
   int[] depth = kinect.getRawDepth();
  ArrayList blobs = new ArrayList();
  for (int x = 0 ;x < kinect.width; x++) {
    for (int y = 0; y < kinect.height; y++) {       
      int offset =  x + y*kinect.width;
      // Grabbing the raw depth
      int z = depth[offset];
      if (z < 1 || z > 2000) continue;
      boolean foundCloseOneExisting =false;
      for (int i = 0; i < blobs.size(); i++) {
        Blob thisBlob =  (Blob) blobs.get(i);
    
        if (thisBlob.isNear(x, y, z)) {
          thisBlob.addPoint(x, y, z);
          foundCloseOneExisting = true;
          break;
        }
      }
      if (! foundCloseOneExisting) {
        blobs.add(new Blob(x, y, z));
      }
    }
  }

  for (int i = blobs.size()-1; i > -1; i--) {
    Blob thisBlob =  (Blob) blobs.get(i);
    if (thisBlob.isSmall()){
      thisBlob.kill(); 
      continue;
    }
    for (int j = blobs.size()-1; j > -1; j--) {
      if (i == j) continue;
      Blob otherBlob =  (Blob) blobs.get(j);
        if (otherBlob.alive == false) continue;
        if (thisBlob.contains(otherBlob)){
          otherBlob.kill();
        }
    }
  }

  //remove dead ones
 for (int i = blobs.size()-1; i > -1; i--) {
    Blob thisBlob =  (Blob) blobs.get(i);
    if (thisBlob.alive == false){
      blobs.remove(i);
    }
  }
  //show top view
  for (int i = 0; i < blobs.size(); i++) {
    Blob thisBlob =  (Blob) blobs.get(i);
   // println(thisBlob.getAveDepth());
   float r = map(thisBlob.getAveDepth(), 500,1000,0,255);
    stroke(r, 127,127);
    rect(thisBlob.minX, thisBlob.minY, thisBlob.maxX, thisBlob.maxY);
    float minDepthAsY = 480- map(thisBlob.minZ, 0, 1000, 0, 480);
    float maxDepthAsY = 480 - map(thisBlob.maxZ, 0, 1000, 0, 480);
    rect(thisBlob.minX + 640, minDepthAsY, thisBlob.maxX + 640, maxDepthAsY);
  }
}



class Blob {
  int minX;
  int minY;
  int minZ;
  int maxX;
  int maxY;
  int maxZ;
  boolean alive = true;

  Blob(int _x, int _y, int _z) {
    minX = _x;
    maxX = _x;
    minY = _y;
    maxY = _y;
    minZ = _z;
    maxZ = _z;
  }
  
  boolean isNear(int _x, int _y, int _z) {
    return (_x < maxX+reach && _x >= minX-reach  && _y <= maxY + reach && _y >= minY -reach   && _z <= maxZ + reach*reachZMultiplier && _z >= minZ -reach*reachZMultiplier) ;
  }

  void addPoint(int _x, int _y, int _z) {
    if (_x >= maxX) maxX = _x;
    if (_x <= minX) minX = _x;
    if (_y > maxY) maxY = _y;
    if (_y <= minY) minY = _y;
    if (_z >= maxZ) maxZ = _z;
    if (_z <= minZ) minZ = _z;
  }
  
  int getAveDepth(){
    return(maxZ - minZ)/2 + minZ;
  }
  
  boolean isSmall(){
    return((maxX - minX)*(maxY-minY) < minSize);
  }
  
  boolean contains(Blob otherBlob){
    return( otherBlob.maxX <= maxX && otherBlob.minX >= minX && otherBlob.maxY <= maxY && otherBlob.minY >= minY && otherBlob.maxZ <= maxZ && otherBlob.minZ >= minZ);
  }
  
  void kill(){
    alive = false;
  }
}

void keyPressed(){
  if(key== 'S'){
    minSize++;
  }else if (key == 's'){
    minSize--;
  }else if(key== 'R'){
    reach++;
  }else if (key == 'r'){
    reach--;
  }else if(key== 'M'){
   reachZMultiplier++;
  }else if (key == 'm'){
    reachZMultiplier--;
  }else if (key == 'b'){
    //you need this for bug in model 1473 model kinect because of bug in library
    kinect = new Kinect(this);
  kinect.initDepth();
  kinect.initVideo();
  kinect.enableMirror(true);
  }
  println(reach, reachZMultiplier, minSize);
}