Introduction To Computational Media (ICM) : Week 5

External Libraries

Processing offers us the ability to use external libraries (created in Java) that are crafted in a special manner.

A couple of interesting and useful ones are:
  • Video: video capture and playback.
  • Serial: reading and writing data through the serial port.
  • Net: reading and writing data over the network.
  • OpenGL: highly optimized graphics API.
  • Sonia: advanced audio capabilities.
  • More at:

  • To utilize an external library, you first need to install that library in Processing.

    The library should be placed in the Processing Application's "libraries" folder.

    To use the library in your Processing application, you need to create an "import" statement which specifies the library name including any "packages" that library is in. (The easy way to do this is through Processing's "Import Library" command under the "Sketch" menu.

    The import statement for the video library:
    The * means import all of the Classes defined within that Library.

    Gimme Eyes

    Video Capture

    Processing has an external library for capturing video from live cameras (firewire, composite or s-video and DV, basically anythiing that QuickTime supports).

    (If you are working on a pc you may need this additional driver for your camera: You may also need to download QuickTime and if you have a choice, do the "Custom" install and select "QuickTime for Java" support.)

    Here is an example of displaying the video from a camera:
                // Import the Library
                // Create an Object of type Capture
                Capture myCapture; 
                void setup() 
                  size(320, 240);
                  // List the available video capture devices    
                  // Initialize the capture object, give it "this", the width you would like, the height and the framerate.
                  myCapture = new Capture(this, width, height, 30); 
                  // Another version of the capture object initialization, this time specifying the capture device
                  myCapture = new Capture(this, "IIDC FireWire Video", width, height, 30);
                // A function that get's called when a new frame is available 
                void captureEvent(Capture myCapture) 
                  // Read the captured frame
                void draw() 
                    // Draw the captured frame as an image
                    image(myCapture, 0, 0); 

    Pixels, Pixels and More Pixels

    Processing offers the ability to get at the array of pixels that make up what is display on the screen, an image, or video frame.

    Here is a quick example that changes an image from RGB to Black and White and back again when the mouse is clicked:
                PImage img;
                boolean showBandW = false;
                void setup()
                  img = loadImage("example_image.jpg");
                  println("Our image is " + img.width + " pixels wide");
                  println("Our image is " + img.height + " pixels high");
                void draw()
                  if (!showBandW)
                    image(img,0,0);  // Just to draw the image normally
                    loadPixels();  // Tell Processing that we are going to access the pixels of the screen
                    // Create a loop within a loop to get at all of the columns and rows of pixels
                    for (int x = 0; x < img.width; x++)
                      for (int y = 0; y < img.height; y++)
                        int currentPixel = y * img.width + x;
                        float currentPixelBrightness = brightness(img.pixels[currentPixel]);  // Pull out the brightness of that pixel
                        color c = color(currentPixelBrightness); // Create a color using that, black and white because not three values
                        //img.pixels[currentPixel] = c;  // Set the pixel of the image to that value
                        pixels[currentPixel] = c;  // Set the pixel of the screen to that value
                    //image(img,0,0);  // Display our image
                    updatePixels();  // Tell Processing we are done with the pixel manipulation and we want to display our work
                void mousePressed()
                  showBandW = !showBandW;
    Here is the Image
    See it in action

    The pixels array allows you to access and set what is displayed on the screen.

    The pixles array of a PImage object allows you to access and set the pixels of the image.

    The pixles array of a Capture object allows you to access and set the pixels of a video frame.

    Video Tracking

    One interesting thing that can be done with video capture is to use it as a means to track obects through space.

    Here is an example (courtesy of DanO):
                Capture video; 
                float targetRed = 255.0; //set some numbers for the target you are chasing
                float targetGreen = 0.0;
                float targetBlue = 0.0;
                void setup()
                  video = new Capture(this, 200, 200, 12); //initiate the video, resolution and frame rate
                  size(320, 240); //give you Processing window a size
                void captureEvent(Capture camera) 
                void draw()
                  float worldRecord = 1000.0; //intialize the worldrecord
                  int xFound = 0; // initialize the location of the red tracking ball
                  int yFound = 0;
                  for(int j=0; j < video.height; j=j+1) { //for each row
                    for(int i=0; i < video.width; i=i+1) { //for each column
                      //get the color of this pixels
                      //find pixel in linear array using formula: pos = row*rowWidth+column
                      color pix = video.pixels[j*video.width+i]; 
                      //find the difference
                      float diff = abs(targetRed - red(pix)) + abs(targetGreen - green(pix)) + abs(targetBlue - blue(pix));
                      if (diff< worldRecord){ // if this is closest to our target color
                        worldRecord = diff;
                        yFound = j; //mark the spot for drawing it later
                        xFound = i;
                  image(video,0,0); //draw the video, this might be optional
                  //after all the pixels have been tested, draw the winner
                  ellipse(xFound, yFound, 10, 10);            
                void mousePressed(){
                  //allow the target color to be changed 
                  color pix = video.pixels[mouseY*video.width+mouseX];
                  targetRed = red(pix); //get the color of the pixel they clicked on
                  targetGreen = green(pix);
                  targetBlue = blue(pix);

    Breakfast Food

    Serial Input and Output

    By now, you should have learned a bit about Serial from the PIC to the PC in Physical Computing. If not, you will shortly ;-).

    Tom Igoe has put together some great pages dedicated to Serial:
  • Serial to the Desktop:
  • More on Serial Communication:

  • We will be using slightly different code than what Tom provides but many of the concepts are the same.

                // Original Example by Dan O'Sullivan (with apologies to Tom Igoe)
                // Heavy mods by Shawn V.
                import processing.serial.*; // The import for the serial library
                Serial myPort;  // The serial port object
                int slide; // A variable to hold our slider value
                int light; // A variable to hold our photocell value
                int button; // A variable to hold our button value
                char DELIMITER = 44;
                char ENDLINE = '\r';
                String savedSerialData = "";
                void setup() 
                  // List all the available serial ports: 
                  // I know that the first port in the serial list on my mac 
                  // is always my  Keyspan adaptor, so I open Serial.list()[0]. 
                  // Open whatever port is the one you're using. 
                  myPort = new Serial(this, Serial.list()[1], 9600); 
                  //prime the pump in case your microcontroller is stuck in serin 
                void draw() 
                  fill(255,light,255);  // Use the photocell value to set our Blue fill
                  if (button == 1) // If our button is pressed
                    ellipse(slide,100,10,10);  // Draw an elipse based on the slider value
                  else  // If our button is not pressed
                    rect(slide,100,10,10);  // Draw a rect based on the slider value
                // Our callback function for serial, gets executed whenever serial data is available
                void serialEvent(Serial p) 
                  String serialData = myPort.readStringUntil(ENDLINE);
                  if (serialData != null)
                    String[] parsedSerialData = split(serialData, DELIMITER);
                    int[] serialDataInts =  int(parsedSerialData);
                    if (serialDataInts.length >= 3)
                      light = serialDataInts[0];
                      slide = serialDataInts[1];
                      button = serialDataInts[2];
                    int servoRange = 250-65;
                    float percentageAcross = mouseX/float(width);
                    int servoOut = int(servoRange*percentageAcross) + 65;
    The code on our PIC (see the PIC BASIC Text Style)

    Serial on the Mac (MacOS X)


    We use rxtx (version 2.1_6) to handle serial I/O, which is included with the Processing release. If this is the first time you're using rxtx, you'll need to make your way to the "libraries" folder, find "serial", and inside, and run macosx_setup.command (double-click it and follow the instructions). This will make sure that things are properly set up (a few permissions need to be changed).