Introduction To Computational Media (ICM) : Week 5

External Libraries

Processing offers us the ability to use external libraries (created in Java) that are crafted in a special manner.

A couple of interesting and useful ones are:

Video: http://www.processing.org/reference/libraries/video/index.html video capture and playback.

Serial: http://www.processing.org/reference/libraries/serial/index.html reading and writing data through the serial port.

Net: http://www.processing.org/reference/libraries/net/index.html reading and writing data over the network.

OpenGL: http://www.processing.org/reference/libraries/opengl/index.html highly optimized graphics API.

Sonia: http://sonia.pitaru.com/ advanced audio capabilities.

More at: http://www.processing.org/reference/libraries/index.html

To utilize an external library, you first need to install that library in Processing.

The library should be placed in the Processing Application's "libraries" folder.

To use the library in your Processing application, you need to create an "import" statement which specifies the library name including any "packages" that library is in. (The easy way to do this is through Processing's "Import Library" command under the "Sketch" menu.

The import statement for the video library:

        import processing.video.*;

The * means import all of the Classes defined within that Library.

Gimme Eyes

Video Capture

Processing has an external library for capturing video from live cameras (firewire, composite or s-video and DV, basically anythiing that QuickTime supports).

(If you are working on a pc you may need this additional driver for your camera: http://www.vdig.com/WinVDIG/. You may also need to download QuickTime http://www.apple.com/quicktime/download/win.html and if you have a choice, do the "Custom" install and select "QuickTime for Java" support.)

Here is an example of displaying the video from a camera:

            // Import the Library
            import processing.video.*; 
            
            // Create an Object of type Capture
            Capture myCapture; 
            
            void setup() 
            { 
              size(320, 240);
              // List the available video capture devices    
              println(Capture.list());
              
              // Initialize the capture object, give it "this", the width you would like, the height and the framerate.
              myCapture = new Capture(this, width, height, 30); 
              
              // Another version of the capture object initialization, this time specifying the capture device
              myCapture = new Capture(this, "IIDC FireWire Video", width, height, 30);
            } 
             
            // A function that get's called when a new frame is available 
            void captureEvent(Capture myCapture) 
            { 
              // Read the captured frame
              myCapture.read(); 
            } 
             
            void draw() 
            {
                // Draw the captured frame as an image
                image(myCapture, 0, 0); 
            }

Pixels, Pixels and More Pixels

Processing offers the ability to get at the array of pixels that make up what is display on the screen, an image, or video frame.

Here is a quick example that changes an image from RGB to Black and White and back again when the mouse is clicked:

            PImage img;
            boolean showBandW = false;
            
            void setup()
            {
              img = loadImage("example_image.jpg");
              println("Our image is " + img.width + " pixels wide");
              println("Our image is " + img.height + " pixels high");
              size(img.width,img.height);
            }
            
            void draw()
            {
              if (!showBandW)
              {
                image(img,0,0);  // Just to draw the image normally
              }
              else
              {  
                loadPixels();  // Tell Processing that we are going to access the pixels of the screen
            
                // Create a loop within a loop to get at all of the columns and rows of pixels
                for (int x = 0; x < img.width; x++)
                {
                  for (int y = 0; y < img.height; y++)
                  {
                    int currentPixel = y * img.width + x;
                    float currentPixelBrightness = brightness(img.pixels[currentPixel]);  // Pull out the brightness of that pixel
                    color c = color(currentPixelBrightness); // Create a color using that, black and white because not three values
                    //img.pixels[currentPixel] = c;  // Set the pixel of the image to that value
                    pixels[currentPixel] = c;  // Set the pixel of the screen to that value
                  }  
                }
            
                //image(img,0,0);  // Display our image
                updatePixels();  // Tell Processing we are done with the pixel manipulation and we want to display our work
              }
            }
            
            void mousePressed()
            {
              showBandW = !showBandW;
            }

Here is the Image
See it in action

The pixels array allows you to access and set what is displayed on the screen.

The pixles array of a PImage object allows you to access and set the pixels of the image.

The pixles array of a Capture object allows you to access and set the pixels of a video frame.

Video Tracking

One interesting thing that can be done with video capture is to use it as a means to track obects through space.

Here is an example (courtesy of DanO):

            import processing.video.*; 
            
            Capture video; 
            
            float targetRed = 255.0; //set some numbers for the target you are chasing
            float targetGreen = 0.0;
            float targetBlue = 0.0;
            
            void setup()
            {
              video = new Capture(this, 200, 200, 12); //initiate the video, resolution and frame rate
              size(320, 240); //give you Processing window a size
            }
            
            void captureEvent(Capture camera) 
            { 
              camera.read(); 
            } 
            
            void draw()
            {
              float worldRecord = 1000.0; //intialize the worldrecord
              int xFound = 0; // initialize the location of the red tracking ball
              int yFound = 0;
            
              for(int j=0; j < video.height; j=j+1) { //for each row
                for(int i=0; i < video.width; i=i+1) { //for each column
                  //get the color of this pixels
                  //find pixel in linear array using formula: pos = row*rowWidth+column
                  color pix = video.pixels[j*video.width+i]; 
                  //find the difference
                  float diff = abs(targetRed - red(pix)) + abs(targetGreen - green(pix)) + abs(targetBlue - blue(pix));
                  if (diff< worldRecord){ // if this is closest to our target color
                    worldRecord = diff;
                    yFound = j; //mark the spot for drawing it later
                    xFound = i;
                  }
                }
              }
              image(video,0,0); //draw the video, this might be optional
              //after all the pixels have been tested, draw the winner
              fill(255,0,0);
              ellipse(xFound, yFound, 10, 10);            
            }
            
            void mousePressed(){
              //allow the target color to be changed 
              color pix = video.pixels[mouseY*video.width+mouseX];
              targetRed = red(pix); //get the color of the pixel they clicked on
              targetGreen = green(pix);
              targetBlue = blue(pix);
            }

Breakfast Food

Serial Input and Output

By now, you should have learned a bit about Serial from the PIC to the PC in Physical Computing. If not, you will shortly ;-).

Tom Igoe has put together some great pages dedicated to Serial:

Serial to the Desktop: http://tigoe.net/pcomp/serial-processing.shtml

More on Serial Communication: http://tigoe.net/pcomp/serialdata.shtml

We will be using slightly different code than what Tom provides but many of the concepts are the same.

        
            // Original Example by Dan O'Sullivan (with apologies to Tom Igoe)
            // Heavy mods by Shawn V.
            
            import processing.serial.*; // The import for the serial library
            
            Serial myPort;  // The serial port object
            
            int slide; // A variable to hold our slider value
            int light; // A variable to hold our photocell value
            int button; // A variable to hold our button value
            
            char DELIMITER = 44;
            char ENDLINE = '\r';
            
            String savedSerialData = "";
            
            void setup() 
            { 
              // List all the available serial ports: 
              println(Serial.list()); 
            
              size(600,600);
            
              // I know that the first port in the serial list on my mac 
              // is always my  Keyspan adaptor, so I open Serial.list()[0]. 
              // Open whatever port is the one you're using. 
              myPort = new Serial(this, Serial.list()[1], 9600); 
            
              //prime the pump in case your microcontroller is stuck in serin 
              myPort.write(65);
            } 
            
            void draw() 
            { 
              background(0);
            
              fill(255,light,255);  // Use the photocell value to set our Blue fill
            
              if (button == 1) // If our button is pressed
              {
                ellipse(slide,100,10,10);  // Draw an elipse based on the slider value
              }
              else  // If our button is not pressed
              {
                rect(slide,100,10,10);  // Draw a rect based on the slider value
              }
            } 
            
            // Our callback function for serial, gets executed whenever serial data is available
            void serialEvent(Serial p) 
            { 
              String serialData = myPort.readStringUntil(ENDLINE);
            
              if (serialData != null)
              {
                //println(serialData);
                String[] parsedSerialData = split(serialData, DELIMITER);
            
                //println(parsedSerialData.length);
                int[] serialDataInts =  int(parsedSerialData);
                if (serialDataInts.length >= 3)
                {
                  light = serialDataInts[0];
                  slide = serialDataInts[1];
                  button = serialDataInts[2];
                }
                int servoRange = 250-65;
                float percentageAcross = mouseX/float(width);
                int servoOut = int(servoRange*percentageAcross) + 65;
                myPort.write(servoOut);
              }
            
            }

The code on our PIC (see the PIC BASIC Text Style)

Serial on the Mac (MacOS X)

From:http://processing.org/faq/bugs.html#serial

We use rxtx (version 2.1_6) to handle serial I/O, which is included with the Processing release. If this is the first time you're using rxtx, you'll need to make your way to the "libraries" folder, find "serial", and inside, and run macosx_setup.command (double-click it and follow the instructions). This will make sure that things are properly set up (a few permissions need to be changed).