Introduction To Computational Media (ICM) : Week 6

External Libraries

Processing offers us the ability to use external libraries (created in Java) that are crafted in a special manner.

A couple of interesting and useful ones are:
  • Video: http://www.processing.org/reference/libraries/video/index.html video capture and playback.
  • Serial: http://www.processing.org/reference/libraries/serial/index.html reading and writing data through the serial port.
  • Net: http://www.processing.org/reference/libraries/net/index.html reading and writing data over the network.
  • OpenGL: http://www.processing.org/reference/libraries/opengl/index.html highly optimized graphics API.
  • Sonia: http://sonia.pitaru.com/ advanced audio capabilities.
  • More at: http://www.processing.org/reference/libraries/index.html


  • To utilize an external library, you first need to install that library in Processing.

    The library should be placed in the Processing Application's "libraries" folder.

    To use the library in your Processing application, you need to create an "import" statement which specifies the library name including any "packages" that library is in. (The easy way to do this is through Processing's "Import Library" command under the "Sketch" menu.

    The import statement for the video library:
            import processing.video.*;
            
    The * means import all of the Classes defined within that Library.

    Gimme Eyes

    Video Capture

    Processing has an external library for capturing video from live cameras (firewire, composite or s-video and DV, basically anythiing that QuickTime supports).

    (If you are working on a pc you may need this additional driver for your camera: http://www.vdig.com/WinVDIG/. You may also need to download QuickTime http://www.apple.com/quicktime/download/win.html and if you have a choice, do the "Custom" install and select "QuickTime for Java" support.)

    Here is an example of displaying the video from a camera:
                // Import the Library
                import processing.video.*; 
                
                // Create an Object of type Capture
                Capture myCapture; 
                
                void setup() 
                { 
                  size(320, 240);
                  // List the available video capture devices    
                  println(Capture.list());
                  
                  // Initialize the capture object, give it "this", the width you would like, the height and the framerate.
                  //myCapture = new Capture(this, width, height, 30); 
                  
                  // Another version of the capture object initialization, this time specifying the capture device
                  myCapture = new Capture(this, "IIDC FireWire Video", width, height, 30);
                } 
                 
                // A function that get's called when a new frame is available 
                void captureEvent(Capture myCapture) 
                { 
                  // Read the captured frame
                  myCapture.read(); 
                } 
                 
                void draw() 
                {
                    // Draw the captured frame as an image
                    image(myCapture, 0, 0); 
                } 
            
    As you can see, capturing video is taken care of by the "Capture" class/object and drawing it to screen is done the same way images are drawn to screen.

    Filtering/Tinting Images/Video

    Processing has a couple of built in functions for filtering and tinting images/video. Here is an example (click the mouse to see the different effects on the video).
                // Import the Library
                import processing.video.*; 
                
                // Create an Object of type Capture
                Capture myCapture; 
    
                int clickCounter = 0;
                int currentMode = 0;
                int numModes = 6;
                            
                void setup() 
                { 
                  size(320, 240);
                  // List the available video capture devices    
                  println(Capture.list());
                  
                  // Initialize the capture object, give it "this", the width you would like, the height and the framerate.
                  //myCapture = new Capture(this, width, height, 30); 
                  
                  // Another version of the capture object initialization, this time specifying the capture device
                  myCapture = new Capture(this, "IIDC FireWire Video", width, height, 30);
                } 
                 
                // A function that get's called when a new frame is available 
                void captureEvent(Capture myCapture) 
                { 
                  // Read the captured frame
                  myCapture.read(); 
                } 
                 
                void draw() 
                {
                    // Draw the captured frame as an image
                    image(myCapture, 0, 0); 
                    
                    // Check the currentMode variable and do the appropriate effect
                    if (currentMode == 0)
                    {
                      // Remove the tint so the image draws normally
                      noTint();
                    }
                    else if (currentMode == 1)
                    {
                      filter(THRESHOLD); // Add the threshold filter to the image
                    }
                    else if (currentMode == 2)
                    {
                      filter(BLUR); // Add the blur filter to the image                  
                    }
                    else if (currentMode == 3)
                    {
                      filter(INVERT); // Add the invert filter to the image                  
                    }
                    else if (currentMode == 4)
                    {
                      filter(POSTERIZE,5); // Add the posterize filter to the image                  
                    }
                    else if (currentMode == 5)
                    {
                       color c_blue = color(0,0,255); 
                       tint(c_blue); // Call the tint filter (won't see until the next time through)
                    }
                } 
                
                void mousePressed()
                {
                  clickCounter++;
                  currentMode = clickCounter % numModes;
                  println(currentMode);
                }
    		

    Breakfast Food

    Serial Input and Output

    By now, you should have learned a bit about Serial from Arduino to the PC in Physical Computing. If not, you will shortly ;-).

    Tom Igoe has put together some great pages dedicated to Serial:
  • Serial to the Desktop
  • More on Serial Communication: interpreting the bytes

  • DanO also has several examples:
  • ICMSerial


  • We will be using slightly different code than what they provide but the concepts are the same.

            
                // Original Example by Dan O'Sullivan - Heavy mods by Shawn V.
    			import processing.serial.*;  // Import the serial library
    			
    			Serial port;       // The serial port
    			
    			int bgColor;       // Background color
    			
    			int shape;
    			int shapeColor;    // Fill color
    			
    			int xpos, ypos;   //  position of the shape
    			
    			void setup() 
    			{
    			  size(256, 256);
    			  noStroke();  // No border on the next thing drawn
    			
    			  // Print a list of the serial ports, for debugging purposes to find out what your ports are called:
    			  println(Serial.list());
    			
    			  port = new Serial(this, Serial.list()[2], 9600);  //you can pull the name out of the list
    			  //port = new Serial(this, "COM11", 9600);  //or you can just specify it
    			  
    			  port.write(65);    // Send a capital A in case the microcontroller is waiting to hear from you (65 is the ASCII value of A)
    			  println("Serial Should Be Ready!");
    			  
    			  bgColor = 0;
    			  shapeColor = 255;
    			  xpos = 0;
    			  ypos = 0;
    			}
    			
    			void draw() 
    			{
    			  background(bgColor);
    			  fill(shapeColor,0,0);
    			
    			  // Draw the shape, either an ellipse or a rectangle
    			  if (shape == 0)
    			  {
    				ellipse(xpos, ypos, 20, 20);
    			  }
    			  else
    			  {
    				rect(xpos, ypos, 20, 20);
    			  }
    			
    			}
    			
    			void serialEvent(Serial port) 
    			{
    			  /*  Read Byte Example  // byte is a value between 0 and 255, it is the most basic way things are transmitted over serial
    			  while(port.available() > 0) // Gives us number of bytes available
    			  {
    				byte input = port.read(); // Read a single byte
    				println(input); // Here we would do something with that byte
    			  }
    			  port.write(65); // Tell the board we are ready for the next thing
    			  */
    			  
    			  /* Read Char Example
    			  while(port.available() > 0)
    			  {
    				char input = port.readChar(); // Reads a single byte interpretted as a character (ASCII)
    				println(input);
    			  }
    			  port.write('A');
    			  */
    				
    			  String input = port.readStringUntil(13);  // Read from serial as a string until the byte you are looking for
    			  // Requires that you return (Ascii 13) at the end of your transmission
    			  // Returns null if it doesn't find the byte
    			  if (input != null)
    			  {
    				 //println("Raw Input: " + input);
    				 String[] parts = input.split(",");  // If you put commas (Ascii 44) between things in your transmission
    				 
    				 // Gives us an array, parts that we can use to drive our variables
    				 
    				 xpos = int(parts[0]);  // The first thing sent is our xpos\
    				 //scaleIt(int highOutput, int lowOutput, int lowInput, int highInput, int currentInput)
    				 xpos = scaleIt(width,0,400,625,xpos);  //400 and 625 were arrived at  by looking at readings from the sensor
    				  
    				 
    				 ypos =  int(parts[1]);  // The second thing sent is our ypos
    				 ypos = scaleIt(height,0,400,625,ypos);  
    			
    				 bgColor =  int(parts[2]); // The third thing sent is our bgColor
    				 bgColor = scaleIt(255,0,500,515,bgColor);  
    				 
    				 shapeColor =  int(parts[3]); // The fourth thing sent is our shape color
    				 shapeColor = scaleIt(255,0,300,700,shapeColor);  
    				 
    				 shape = int(parts[4]);  // The fifth thing sent is our shape
    				 
    				 // Let's see what we got
    				 //println("Parsed and Scaled x:" + xpos + " y:" + ypos + " bgColor:" + bgColor + " shapeColor:" + shapeColor + " shape:" + shape);
    				 
    				 port.write(65);    // Send a capital A (ascii 65) or any other arbitrary character in case the microcontroller is waiting to hear from you
    			   }
    			}
    			
    			
    			/*
    			This function takes the numbers that come in from you sensors
    			and scales them into a range that processing can use, for 
    			instance 0-255 for colors
    			*/
    			int  scaleIt(int highOutput, int lowOutput, int lowInput, int highInput, int currentInput)
    			{
    			  int inputRange = highInput - lowInput;  // Get the input range
    			  int outputRange = highOutput - lowOutput; // Get the output range  
    				
    			  // Make sure current input is within the high and low input ranges
    			  currentInput = min(currentInput,highInput);  // protect against high rouge inputs by taking the minimum between the current and the highest allowable
    			  currentInput = max(currentInput,lowInput);  //protect against low rouge inputs by taking the maximum between the current and the lowest allowable
    			  
    			  int currentPlaceInInputRange = currentInput - lowInput;  // How far in the range is it.
    			  float percentagePlaceInputRange = (float)currentPlaceInInputRange/(float)inputRange;  // Get the percentage (not 100 percent, between 0 and 1)
    			
    			  int placeInOutputRange = (int) (percentagePlaceInputRange * outputRange);
    			  println("High Output: " + highOutput + " Low Output: " + lowOutput + " Low Input: " + lowInput + " High Input: " + highInput + " Current Input: " + currentInput);
    			  println("Input Range: " + inputRange + " Output Range: " + outputRange + " Current Input: " + currentInput + " Current Place: " + currentPlaceInInputRange +  " Percentage: " + percentagePlaceInputRange + " Place In Output Range: " + placeInOutputRange);
    			  
    			  return placeInOutputRange;
    			}
    						
            
    The code on our Arduino (See Arduino (Text Style))

    Serial Reference:

    Serial Libraries - Check this for possible gotchas getting serial to work on your machine.