Sharing

Whether it is motivated by a drive to commune with other lonely souls, or a drive to externalize their feelings or by memes which have taken over their minds, people want to share their experience.

• Computer mediated communication needs work. It is currently slow, narrow and filters out the most interesting bits.
• Most of the time we make a connection to third party (client to server) rather than directly to another person (peer to peer)
• HTTP (web) style networking is always instigated from the client and is thus slow and better for asynchronous communication. In processing we use the [loadstring] function for this.
• Socket style networking is can be two way and is much faster. There is a [net library] for this doing this style of networking in processing.

The loadstrings Command, HTTP connections in Processing,

• Processing command that can similate typing a command in the location box of your browser
• It returns the raw text (usually formatted as html or xml) that would otherwise be interpreted by your browser
• For some reason loadstrings is returned as an array of of text strings instead of one big string. I would use the join function to sew that array back into one big string. String oneBigString = join(arrayOfStrings,"");
• Loadstrings will also works for getting text locally from a file if you specify a path name instead of a url.
• Communication over the internet is asynchronous, meaning you don't know exactly when the response will come. Loadstrings uses the "blocking" technique of dealing with asynch meaning it stops you programming from doing anything else until it gets a response. If the response takes a long time, your program might feel a little chuncky. You can use Shiffman's new library to get around this. Or you can wait until next week we will solve the same asynchrony problem in Serial.

The revolutionary query string

• When you ask for web page with loadstrings, it is not much more special than changing the channel on your tv.
• There is also room at the end of the url (after the a "?") for giving (not just getting) information to the server. This text after the "?" is called the query string.
• You can think of this as sending parameters variable along with the url to a function that lives on the server.
• These parameters are usually organized into "name value pairs" like this ...?nameofVar1=valueOfVar1&nameofVar2=valueOfVar2&nameofVar3=valueOfVar3
• HTML forms are the usual device for generating and formating these variables at the end of the url but we will be using processing (loadstrings) instead:

  String url = "http://xml.weather.yahoo.com/forecastrss?p=" + zip;
  String[] lines = loadStrings(url);

• Tacking this information on to the end of the URL is called the "GET" method. You may also hear about another method "POST" where you can send the extra info in a bigger more hidden separate package.

Text In and Text Out

A lot of the programming that people do these days involves communication between software usually over the internet. At the end of the day this is usually done by sending text strings back and forth. Learning how to put together (contatinate) and pull apart (parse) strings is very important.

• Text is an easier target than user interface because it stands still.
• You have probably already used the "+" operator which in the context of strings ceases to be a math operator but instead acts as glue between bits of text.
• As it turns out, the String type that we have been using was secretly the Java String type so you will have all the fuctionality of that class.
• This is the beginning of the integration of Java with processing.
• Check out the documentation for the java [String] class which has a lot of extra functionality that you can use
• Processing has a command called splitStrings() or you can use the java split() that is useful for cutting (parsing) up strings.
• Also of particular interest are "indexOf()" , "length()" and "substring()"
• Most of your work will be using repeat loops to go letter by letter, word by word, line by line, or tag by tag through a body of text looking for stuff.

Serverside Programming

• When you send parameters with the url, usually it is a program on the server that recieves them.
• Serverside programs are important for allowing something of the user to remembered and possibly passed on to other users.
• Basicallly anytime you want information to persist across space or time, you will need to use serverside programs to recieve and record that information and act as central point between users or between sessions of the same user.
• More often they are used to produce web pages on the fly customized slightly by the parameters sent in at the end of the url.
• You can write these serverside programs in many languages like Java (servlets), Perl, Php etc...

Using Other People's Server Side Programming

• In order to avoid having to do serverside programming you can be a parasite on an existing engine like googe, dictionary basically any web page.
• While using these other services saves you from learning to do server side programming, you do have to be pretty clever about reverse engineering their protocol for input on the query string and formatting for html or xml output.
• Scraping HTML.
  • You have to first reverse engineer the protocol they use in bringing up the web page.
  • While using the site you would like to scrape, carefully watch watch happens in the location box and look for patterns in the query string.
  • The next step is to examine what comes back from the web. You will have to use the "view source" menu item.
  • Unfortunately HTML is meant to talk to your eyes so picking out the bit you want out the mess of html can be a challenge.
• XML
  • Often you can find server side programs that give you XML, such as rss feeds, instead of HTML. XML is formatted in a way that is deliberately designed for other software to read.
  • Check the [main icm page] for a list of XML feeds.
  • You can parse through this using your own ordinary processing code (mostly while loops) looking for stuff. Sometimes you might need recursion dig down into it so you migh want to use some libraires.
    • You can parse through the code using Shiffman's easy [1] XML parsing library.
    • You can use a generic [XML parser] to pre parse the tags for you.
    • You can use a library that is specific to the XML feed you are using. (eg [google]).

Security Sandbox

• When you are operating in the Processing you have a lot of freedom to reach out into the machine or into the network.
• You are a good person but not all applets can be trusted so your program's freedom is limited when running in a browser.
• You have already encountered this with serial and video.
• You are not allowed to make a network connection back to a machine other than the one that served you up.
• This means that your loadstrings command will only work with urls on the itp server.
• For now just work within Processing
• Work arounds are basically getting a proxy program on the itp server to relay your requests

String proxyURL  = "http://itp.nyu.edu/proxy/proxy.php?url=";
String myURL = "http://www.cnn.com";
String[] arrayOfLines = loadStrings(proxyURL  + myURL);    //combine them, basically send your url as querystring to the proxy 

• or signing your applet. *Here is more [info].

Getting Image of Stocks from cnn.com (HTML)

PImage stockIndicator;
void setup(){

  String myURL = "http://marketwatch.nytimes.com/custom/nyt-com/html-usmarkets-quote.asp?sid=1643";
  String[] arrayOfLines = loadStrings(myURL);
  String oneLongString = join(arrayOfLines, "");
  //narrow in on the whole page to something close by
  int near = oneLongString.indexOf("Dow Jones Industrial Average");
  //the html comign back gets broken up in to lines but we want it in one long string, the join command does this
   String startingLandMark = "img src=";
  //String startingLandMark = "DOW</a><div class=\"cnn4pxTpad\"><img src=";  
  //this always comes before the thing I want, notice I used the escape quote \" to include a quote in the quoted
  int start = oneLongString.indexOf(startingLandMark,near) + startingLandMark.length();  
  //find the first occurance of this landmark and then march forward past the end of the landmark
  int end = oneLongString.indexOf(">",start);  
  //look for the end landmark in this case a quote,  notice I used the escape quote \" to include a quote in the quoted
  //also notice that I used a variation of indexOf that starts looking not at the beginning of the string 
  //but a  point I specify, in this case the the beginning of what I already found
  String imageURL = "http://marketwatch.nytimes.com/custom/nyt-com/" + oneLongString.substring(start+1,end-1); //+1 -1 to loose the quotation marks
 
  println(imageURL);
  stockIndicator = loadImage(imageURL);
}

void draw(){
  image(stockIndicator,width/2,height/2);
}

Getting Dog Pictures from google (HTML)

//Dano's example, with slight adjustments.
int MAX_IN_ARRAY = 1000;
PImage[] images = new PImage[MAX_IN_ARRAY];
int numberFound = 0;

void setup(){
  size(800,800);
  String proxyURL  = "http://itp.nyu.edu/icm/proxy/proxy.php?url=";
  //this is a relay station on itp that passes your request on, this allows your applet to work in a browser
  //I told you not to worry about this and just work in proccessing            
  String myURL = "http://images.google.com/images?hl=en&q=dog";
  //String[] arrayOfLines = loadStrings(myURL);
  String[] arrayOfLines = loadStrings(proxyURL  + myURL);
  String oneLongString = join(arrayOfLines, "");

  //the html comign back gets broken up in to lines but we want it in one long string, the join command does this
  String startingLandMark = ":\",\"http:" ; //"images?q=tbn:";  
  int startLookingAt = 0;  //this is like our cursor to start looking after the last place we found somthing

  boolean stopIt = false;
  while( !stopIt ){  //different kind of repeat loop that goes for an undetermined number of interations
    //this always comes before the thing I want, notice I used the escape quote \" to include a quote in the quoted
    int start = oneLongString.indexOf(startingLandMark,startLookingAt); //you want to include the landmark  so don't do this + startingLandMark.length();  
    //find the first occurance of this landmark and then march forward past the end of the landmark
    //also notice that I used a variation of indexOf that starts looking not at the beginning of the string 
    //instead at a  point I specify, in this case after what I already found

    if (start == -1) {
      //this means it hasn't found anything, so set our variable and our WHILE loop will stop
      stopIt = true;
    } else {
      start = start + startingLandMark.length() ; //jump to the end of the startlandmark
      int end = oneLongString.indexOf("\"",start  );  
      //look for the first end landmark, in this case a space, after the starting landmark
      String imageURL = "http:" +  oneLongString.substring(start,end); //add google's default path
      println("image... " + imageURL);
      images[numberFound] = loadImage(proxyURL + imageURL);
      numberFound++;
      startLookingAt = end + 1;
      if (numberFound >= MAX_IN_ARRAY ) {
        //don't overpack the array
        stopIt = true; //stops the repeat loop;
      }
    }
  }

}

void draw(){
  float angleIncrements = 2*PI/numberFound;  //angle units between each picture, in radians 2*PI = 360 degrees
  int radius = 150;
  for(int i = 0; i < numberFound; i++){
    float thisAngle = i* angleIncrements;
    float x = radius*cos(thisAngle)  + width/2;
    float y = radius*sin(thisAngle)  + height/2;
    image(images[i],x,y,50,50);
  }

}

Getting Weather from Yahoo (XML)

// Daniel Shiffman
// ICM, 2005
// Parsing Yahoo's XML Weather Feed

PFont f;

String weather = "";
String[] zips = {"10003","21209","90210"};
int temp;
int counter = 0;

void setup() {
  size(200,200);
  // Call our new getWeather function!
  getWeather(zips[counter]);
  f = loadFont("Georgia-Bold-16.vlw");
  println(temp);
}

void draw() {
  background(100);
  textFont(f);
  textMode(SCREEN);
  fill(255);
  // Display all the stuff we want to display
  text(zips[counter],10,160);
  text(weather,10,90);
  text(temp,10,40);
  text("Click to change zip.",10,180);


  // Draw a little thermometer based on the temperature
  noStroke();
  fill(200);
  rect(10,50,temp*2+random(0,5),20);

}

void mousePressed() {
  // When the mouse is clicked, increment the counter and get the weather at the next zip code
  counter = (counter + 1) % zips.length;
  getWeather(zips[counter]);
}

void getWeather(String zip) {
  //Get all the HTML/XML source code into an array of strings (each line is one element in the array)
  String url = "http://itp.nyu.edu/icm/proxy/proxy.php?url=http://xml.weather.yahoo.com/forecastrss?p=" + zip;
  String[] lines = loadStrings(url);

  String xml = join(lines, " "); // Get rid of the array and make it one very long String

  // The manual way to concatenate
  // String xml = "";
  // for (int i = 0; i < lines.length; i++) {
  //   xml += lines[i];
  // }

  // Searching for weather condition
  String lookfor = "<yweather:condition text=\"";
  String end = "\"";
  weather = giveMe(xml,lookfor,end);

  // Searching for temperature
  lookfor = "temp=\"";
  temp = int(giveMe(xml,lookfor,end));

}

// A function that returns a substring between two substrings
String giveMe(String s, String before, String after) {
  String found = "";
  int start = s.indexOf(before);  // Find the index of the beginning tag
  if (start == -1) return ""; // If we don't find anything, send back a blank String
  start += before.length(); // Move to the end of the beginning tag
  int end = s.indexOf(after,start); // Find the index of the end tag
  if (end == -1) return ""; // If we don't find the end tag, send back a blank String
  return s.substring(start,end); // Return the text in between
}