It is easy to put images in  processing.  There are two commands, loadImage (in setup) and image (in draw) that works pretty much like the rect and ellipse command but draws an image.  This is the same pattern as putting sound into your program.  Load the sound in setup() and use it in draw().

The problem with the image command, is that it is pretty much take it or leave it.  You can’t change or analyze the image.  This week we are going to dive into the pixels of the images coming in from a camera to analyze and change the image.

Images as Arrays

Images are stored as a collection of pixels.  As we learned last week, if your program wants to wants to change those images or analyze the pixlels we are going to have to put them into variables.   You would need a lot of variables to have one for each pixel of even a small picture.  To the rescue there is a special type of variable called an array that has many slots in it.  We can address the individual slots in the array by number (starting with zero).  It is sort of like this pill box named myvar with the number 169 stored in the second slot. You use square brackets to say which compartment you want to address. To put 169 in  you would say myvar[1] = 169.  To pull it out you would say int num = myVar[1]  (I know it is weird that the second slot has an index of 1 but you just have to get used to that);


So we store the pixels of an image in a big long array, maybe 500,000 slots.  The pixels for the programs you have made so far were stored for you in a variable that processing makes automatically called pixels.  On a less interesting note, to pull the pixels off the screen into the array you should say loadPixels() and to pull the pixels out of the array back onto the screen you should say updatePixels().  This program sets those pixels.

Arrays and For Loops

Now we don’t need a million variables for a million pixels because we can just have an array variable with a million slots.  Unfortunately we would still need a million lines of code to pull the pixels out and to set them.  Coming to the rescue is the for loop.  A for loop allows you to repeat a code block (curly brackets) while incrementing a variable to keep track of everything.  The syntax is the word for followed by a parenthesis with three things.  Here is the psuedocode.  Notice  the three parts are separated by semicolons.

for (declare/initialize variable ; limit how big ; how to change variable each loop){
//stuff to repeat

When was the last time you did 5050 things so quickly.  Do you feel the power?  This loop makes an int variable called “i” and puts  zero in it to start.  For some reason programmers love using the variable named i in for loops but you can name it anything.   It will keep doing  the code in the code block over and over until the gets to 5050.  Each time through the variable will get bigger by 1.  i++ is the same thing as saying i = i + 1;

For a video with your favorite explainer check this out.

Going through the pixels coming in from a live camera is the same as looking though the pixels of a still image except you do it 30 time a second.  So the draw loop happens 30 times a second for each frame of video and then a nested for loop happens 320,000 times for each draw loop, once for each pixel.

Live Camera

Look at Pixels in Live Camera


Enter the Kinect

Pixels tell you what color you are but in giving expression to your body you might care more about where your body is rather than what color it is.  The kinect gives you not only an array with colors for each pixel but gives you another array with the depth of each pixel.  Go ahead and install the Kinect.

Install The Kinect Library

  1. In Processing find the menus Sketch>Import Library> Add Library to get a window of available libraries.
  2. Scroll down to “Open Kinect for Processing by Dan Shiffman and Thomas Sanchez” and install it, and close the window.  If you are on a PC you need to run  a program recommended here to get your machine ready.  This is a different and fancier processing library for the PC  that requires that you install something (SDK) from Microsoft.
  3. I usually restart Processing but I think that might be superstition.
  4. Try going the menu Files > Examples… to see the fun stuff you can do.
  5. Import the library into your program by putting these lines at the very top import org.openkinect.freenect.*;  and import org.openkinect.processing.*;   The menu Sketch>Import Library> Open Processing for Kinect will do this for you.

Using the Kinect Library

  1. Make a variable at the top to hold your Kinect object.  Instead of and integer or floating point, it should be of type Kinect.
  2. In setup you have to put a new Kinect object into that variable and then enable a few parts of the Kinect.
  3. In the draw loop go looking through all the the depths of the pixels coming in from the Kinect.
  4. Do something based on some change in depth of the pixels.
  5. Here is the documentation for this library for more info.

Simple Pelvic Thrust

Here is one a little fancier where it has a smaller target within the screen.  For this it need to scan through the pixels in the long array by row and column.  There are variables for the borders of the target within the screen.

Bug Fix in Libary for Kinect model 1473


Vary Position of the Target

Add this little function at the bottom to allow you to change the variables with the keyboard while the program is running.

Overlay Another Image of Depth Pixels

You can make a blank image at the beginning, change the pixels of that depending on the depth and then overlay that on the full color video image you painted at the top of the draw loop.  These are the lines added to make a new image and overlay it:

Here they are integrated into the previous example:

More Than On Off

This is even a little fancier.  It does more than just find whether or not there is a single pixel within that target, it find the average depth within the target and uses that play a sound file a different volumes.

Multiple Targets (cleaned up by making a function)

And finally you can have multiple targets if you farm your code into a resuable custom function that you name and write.  In this case I wrote a new function called getAverageDepth() that encapsulates all the code in that repeat loop.  If I send it different parameters it can do the same repeat loop for different positions.

Find Depth Blogs

Added later after class