Category Archives: Data

Big Data, Little Brain

I really wish my brain could wrap around this idea of Big Data and how exactly to use it. I really do see the importance of it all. Over the next few years I will make it a goal of mine. In the meantime, I can just pretend that I understand it and sound smart to my friends.

I’m still extremely interested in this whole Wikipedia Philosophy thing. After looking more at Wikipedia’s API– the more I understand the importance of APIs as a whole as well as the wealth of information that Wikipedia is.

I decided to make a little mock up of what a possible data visualization of the links to “Philosophy” would be:

philosophy2

 

Would be pretty cool….

In 2 years time, I hope to be able to make the real deal. High hopes! High rewards! Yeeehaw!

 

We Feel…Fine?

Screen Shot 2014-05-04 at 10.03.51 PM

How are you feeling?

Fine is a four-lettered placeholder.

This grabs feelings from the We Feel Fine API. We Feel Fine’s system searches blog entries for occurrences of the phrases “I feel” and “I am feeling.” The latest feeling (sentence) displays in the console and in the sketch – it refreshes every 20 seconds. Usually, it takes up to 40 seconds for a new feeling to surface.

Things to fix:

– Somehow ignore the <feelings></feelings> part of the String, so that only the sentence shows up

– Fit the whole sentence inside of a text box (attempted, but did not work).

Things to add:

– Color change every time the feeling ascribed to the sentence changes. Something that correlates with Goethe’s ideas about psychology in relation to color & emotion.

//most of the code is from the class example with the NYT headlines
String title = "Gathering feelings...";
int lastTimeChecked;
int xpos;

void setup() {
  size(800, 200);
  // textAlign(CENTER,CENTER);
}

void draw() {
  background(0);

  if (millis() - lastTimeChecked > 20000) { 
  //use a timer to check every 20 seconds, because it doesn't update that often anyways
    thread("loadFeelings");  //start a parallel thread to ask for feelings
    lastTimeChecked = millis();   
  }

   /*
   //Attempting to fit all the text within a text box...
   text(title, 200,200, xpos, height/2); //this animation stays smooth because you used a thread
   */

   text(title, xpos, height/2);
   xpos++
    if (xpos > width) xpos = 0;
  }

void loadFeelings() {
  XML root = loadXML("http://api.wefeelfine.org:8080/ShowFeelings?display=xml&returnfields=sentence&limit=1");
  title = root.toString();

  println("check" + title);
}

 

 

Ieva and Emily: Cat Detectives (in training)

import ddf.minim.*;
//import twitter4j-4.0.1.*;
//import twitter4j.api.*;
//import java.util.*;

//audio
AudioPlayer player;
Minim minim;

PImage backimage;
PImage sloth;
PImage catFace;

//Query query;
//Twitter twitter;

void setup() {  

  //Set the size of the stage, and the background to black.
  size(700, 700);
  backimage = loadImage("compcat.jpg");
  sloth = loadImage ("slothallday.jpg");
  catFace = loadImage ("buny_cat.jpg");
  smooth();

 ConfigurationBuilder cb = new ConfigurationBuilder();
 cb.setOAuthConsumerKey("EKTjLWa6EOFXhfthNTP8A");
 cb.setOAuthConsumerSecret("ZTsfvECqYSLF57E3mRLTLm3zevUzLIjEateUSSfPU");
cb.setOAuthAccessToken("329517248-TDTpfMwx6fn1n6O6VpIzmdqpcugbVfJGluNau8Xv");
  cb.setOAuthAccessTokenSecret("DzfsSJmkK1WcqmYAAa2XPoAgaIY8A3H84puznWyeLBrHV");

  //initialize minim
  minim = new Minim(this);
  player = minim.loadFile("catcat.mp3");
  player.play();

  //Make the twitter object and prepare the query
 Twitter twitter = new TwitterFactory(cb.build()).getInstance();
 Query query = new Query("#cat");
 query.setCount(100);

 //Try making the query request.
 try {
 QueryResult result = twitter.search(query);
 List tweets = result.getTweets();

 for (int i = 0; i < tweets.size(); i++) {
 Tweet t = (Tweet) tweets.get(i);
 String user = t.getFromUser();
 String msg = t.getText();
 Date d = t.getCreatedAt();
 println("Tweet by " + user + " at " + d + ": " + msg);

 //Break the tweet into words
 String[] input = msg.split(" ");
 for (int j = 0;  j < input.length; j++) {
 //Put each word into the words ArrayList
 words.add(input[j]);
 }
 };
 }
 catch (TwitterException te) {
 println("Couldn't connect: " + te);
 };
 }

void draw() {
  background(backimage);

  //Draw a word from the list of words that we've built
  //int i = (frameCount % words.size());
  // String word = words.get(i);

  // Put it somewhere random on the stage, with a random size and colour
  fill(255, random(50, 150));
  textSize(random(10, 30));
  //  text(word, random(width), random(height));

  //ieva's dumb people code thinking

 // if tweet contains word cat {
 //   draw (catFace);
 // }

 // if tweet does not contain word cat{
//  background (sloth);
  //}

}

 

Holycrapbooks, Batman!

This week we had to use data. Most of my code is lifted from the example provided in class. I changed it around and tried to make it a bit creepier/more interesting.

    

I compared Isaac Asimov’s Foundation series and J.D. Salinger’s Catcher in the Rye. Both books are cited for inspiring violent behavior and murder in certain individuals (Timothy McVeigh and Mark David Chapman, respectively).  I wanted to compare the words both authors use most often to see if there are any similarities.

I had some trouble removing certain words, so  the way it is in the code is a little slap/dash, because I got overwhelmed with doing something more intricate. I’d like to remove some more of the small words, maybe by taking out every 3-letter word and below.

 

Screen Shot 2014-04-14 at 2.42.07 PM

//asimov and salinger comparison
PImage photo;
String[] catcher; 
String delimiters = " - ,.?!;:[]";
IntDict words;

String[] foundation; 
IntDict panda;

//setup
void setup() {
  size(800, 640);
  photo = loadImage ("owl.jpg");
  String url = "http://www-958.ibm.com/software/analytics/manyeyes/datasets/the-catcher-in-the-rye-full-text/versions/1.txt";
  String[] rawtext = loadStrings(url);
  String everything = join(rawtext, "" );
  catcher = splitTokens(everything, delimiters);
  words = new IntDict();

  for (int i = 0; i < catcher.length; i++) {
    String s = catcher[i].toLowerCase();
//if statement with remove words
    words.increment(s);
  }
  words.sortValuesReverse();
  words.remove("the");
  words.remove("i");
  words.remove("and");
  words.remove("to");
  words.remove("a");
  words.remove("was");
  words.remove("it");
  words.remove("you");
//asimov
  String[] rawfile = loadStrings("asimov.txt");
  String everything2 = join(rawfile, "" );
  foundation = splitTokens(everything2, delimiters);
  panda = new IntDict();

  for (int j = 0; j < foundation.length; j++) {
    String d = foundation[j].toLowerCase();
    panda.increment(d);
  }
  panda.sortValuesReverse();
  panda.remove("the");
  panda.remove("of");
  panda.remove("and");
  panda.remove("a");
  panda.remove("to");
  panda.remove("that");
  panda.remove("you");
  panda.remove("'");

}

//draw
void draw() {
  image(photo, 0, 0);
 // background(255);//no need twice

  int h = 20;
  String[] keys = words.keyArray();

  for (int i = 0; i < height/h; i++) {
    String theword = keys[i];
    int count = words.get(theword);

fill(random (255), random (255), random (255));
    noStroke();
    ellipse(width/4, i*20, count/8, count/8);
    fill(255);
    text(theword, 0, i*h+h/2);
    stroke(0);
  }

//asimov
// Display the text and total times the word appears
  int g = 20;
  String[] keys2 = panda.keyArray();

  for (int j = 0; j < height/g; j++) {
    String theword = keys2[j];
    int count = panda.get(theword);

fill(random (255), random (255), random (255));
    noStroke();
    ellipse(width/4*3, j*20, count/20, count/20);
    fill(255);
    text(theword, width - 70, j*g+g/2);
    stroke(0);
  }

}

 

M&Ms

eminem_mms

For this week’s assignment, I tried to find a fun data set.. I ended up finding some random set a professor took of the color distribution in packs of M&Ms. I am trying to use the average number as the radius of a circle in the window to visualize the data, but right now I’m running into a null pointer exception that I’m trying to fix.

//Colors of milk chocolate M&Ms in the fun size packages
//collected by Todd Swanson and his statistics students in 2001.
//http://www.math.hope.edu/swanson/data/mms.txt

String [][] data;
float brown, red, yellow, blue, green, orange;
MandM brownM, redM, yellowM, blueM, greenM, orangeM;
float colorXpos, colorYpos;
float brownSum, redSum, yellowSum, blueSum, greenSum, orangeSum;
int xPos, yPos;

void setup() {
  //the basics
  size(640, 280);
  
  String line[] = loadStrings("mms.csv");
  data = new String [21][6];
  for (int row=0; row < 21; row++) {
    String [] temp = new String [6];
    temp= split(line[row], ',');
    for (int column=0; column < 6; column++){
        data[row][column]= temp[column];
    }
   }
 
 //Calculating the sum
  for (int row=0; row < 21; row++) {
      brownSum += parseInt(data[row][0]);
      redSum += parseInt(data[row][1]);
      yellowSum += parseInt(data[row][2]);
      blueSum += parseInt(data[row][3]);
      greenSum += parseInt(data[row][4]);
      orangeSum += parseInt(data[row][5]);
   }

 //To visualize data: average is the radius of ellipse
 //Already know there are 21 total rows
    brown = brownSum/21;
    red = redSum/21;
    yellow = yellowSum/21;
    blue = blueSum/21;
    green = greenSum/21;
    orange = orangeSum/21;
    
    for (int i = 0; i < 6; i ++){
       brownM =new MandM (random(0,width), random(0,height), brown*15);
       redM =new MandM (random(0,width), random(0,height), red*15);
       yellowM =new MandM (random(0,width), random(0,height), yellow*15);
       blueM =new MandM (random(0,width), random(0,height), blue*15);
       greenM =new MandM (random(0,width), random(0,height), green*15);
       orangeM =new MandM (random(0,width), random(0,height), orange*15);
   }
    
 

}
 
void draw () {
    background(255);
   
    MandM [] funSize = new MandM [6];
    funSize[0] = brownM;
    funSize[1] = redM;
    funSize[2] = yellowM;
    funSize[3] = blueM;
    funSize[4] = greenM;
    funSize[5] = orangeM;
   
    for (int i=0; i<6; i++){
           if (i==0){
          fill(110,70,0);
          
        }
        if (i==1){
          fill(255,0,0);
        }
        if (i==2){
          fill(255,255,0);
        }
        if (i==3){
          fill(0,0,255);
        }
        if (i==4){
          fill(0,255,0);
        }
        if (i==5){
          fill(255,150,0);
        }
      funSize[i].drawMandM();
  }
   
}

///////
class MandM {
  
  float xPos, yPos, radius;
 
  MandM(float x, float y, float radius){
   this.xPos = x;
   this.yPos = y;
   this.radius = radius;
    
  }
  
  
  void drawMandM(){
    for(int i = 0; i < 6; i ++){
        noStroke();
        smooth();
        ellipse(xPos, yPos, radius, radius);
        noLoop();
     }  
  }
}

 

Madlibs!

For my data assignment, I’ve decided to create a madlibs game. PS. You can’t use the “backspace”, so be sure to check your spelling!

madlibs

Here’s the link to download:

https://drive.google.com/folderview?id=0B9jm5VZVC2_odFJUTWdZQlMtcFU&usp=sharing

import java.io.*;

StringDict store_input;
PFont type; 
PFont read; 
int word_count;
boolean search_switch;
PImage permanent;
PImage note;

// Variable to store text currently being typed
String typing = "";

// Variable to store saved text when return is hit
String saved = "";

String [] keys = {
  "",
  "adjective 1", 
  "adjective 2", 
  "noun 1", 
  "noun 2", 
  "noun 3", 
  "noun 4",
  "adjective 3", 
  "adjective 4", 
  "adjective 5", 
  "verb 1 (in -ing) form", 
  "verb 2", 
  "adverb 1", 
  "adjective 5", 
  "pronoun 1", 
  "verb 2", 
  "adjective 6", 
  "noun 5", 
  "noun 6", 
  "adjective 8", 
  "adjective 9", 
  "noun 7", 
  "noun 8", 
  "adjective 11", 
  "adjective 12", 
  "your name", 
  "adjective 13", 
  "noun 9",
  "noun 10"

};

void setup() {
  size(1000, 700);
  word_count = 0;
  type = loadFont("Cochin-48.vlw");
  read = loadFont("ComicSansMS-24.vlw");
  store_input = new StringDict();
  search_switch = true;
  note = loadImage("note.png");
}

void draw() {

//TEXT ENTRY: 
  if (word_count<29) {
    textFont(type);
    fill(0);
    background(255);
    text("Let's play  madlibs!", 100, 40);
    text("Please enter: "+keys[word_count], 100, 100);    
    text("Your input: "+typing, 100, 200);
    text("You have entered: "+saved, 100,250);

    store_input.set( keys[word_count],saved);
    System.out.println(word_count);

  }//End of text entry

  //Text feedback: 
  else if (word_count >=29) {
    image(note, 0,0);
    fill(0);
    textFont(read);

    text("Dear "+store_input.get(keys[1])+" parents,\n I'm sorry I haven't written to you in a while, I'm writing to you guys to ",50, 30);
   text("tell you that I've officially settled in at NYU! It's been a "+store_input.get(keys[2]), 50, 90);
    text(" week, but I've finally finalized my classes. I'm taking the history of", 50, 120);
    text(store_input.get(keys[3])+" the science of "+store_input.get(keys[4])+" a seminar on "+ store_input.get(keys[5])+" and my elective that" ,50, 150);
    text("is about "+store_input.get(keys[6])+". One of my classes is so "+store_input.get(keys[7])+" because my teacher",50, 175);
    text("is "+store_input.get(keys[8])+". He has this "+store_input.get(keys[9])+" habit of"+store_input.get(keys[10])+" that makes me want to ", 50, 210);
    text(store_input.get(keys[11])+" so "+store_input.get(keys[12])+". My roommate is pretty "+store_input.get(keys[13])+".", 50, 240);
    text("Once, in the middle of the night, "+store_input.get(keys[14])+ " suddenly started to "+store_input.get(keys[15])+".",50, 270);
    text("It was so "+store_input.get(keys[16])+"! But, "+store_input.get(keys[14])+" reassured me everything was ok.", 50, 300);
    text("Also, "+store_input.get(keys[14])+" has this ginormous "+store_input.get(keys[17])+" that blocks the "+store_input.get(keys[18])+", but ultimately", 50, 330);
    text("we get along. New York is the most "+store_input.get(keys[19]),50,360);
    text("city I have ever seen! It's full of "+store_input.get(keys[20])+" people", 50, 390);
    text("and is a huge cultural, transportation, and "+store_input.get(keys[21])+", center!", 50, 420);
    text("There are some downsides living here. For the first time today, a homeless guy", 50, 450);
    text("threw a "+store_input.get(keys[22])+" at me. It was a little "+store_input.get(keys[23]), 50, 480);
    text("but now I'm fine, so don't worry. Anyways, I have to do my", 50, 510);
    text("homework now, so I'll email you like a more "+store_input.get(keys[24])+" person next time.", 50, 540);
    text("Love, "+store_input.get(keys[23])+", your "+store_input.get(keys[26])+" "+store_input.get(keys[27]), 50, 570);
    text("P.S. Do you like the "+store_input.get(keys[28])+" I sent you?\n I got it at the NYU Bookstore!",50, 610);
    if (search_switch){
      queryGoogleImages(store_input.get(keys[28]));
      search_switch = false;
    }
    image(permanent, 800,475, 150, 150);

  }
} // End of Draw

  void keyPressed() {
    // If the return key is pressed, save the String and clear it
    if (key == '\n' ) {
      saved = typing;
      // A String can be cleared by setting it equal to ""
      typing = ""; 
      word_count++;
    } 
    else {
      // Otherwise, concatenate the String
      // Each character typed by the user is added to the end of the String variable.
      typing = typing + key;
    }

  }

  void queryGoogleImages(String _q) {
  String url = "https://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=" + _q;
  //this is a special url for getting google image results as json
  println(url);
  JSONObject result =  loadJSONObject(url);
  //make a network call and get the results back into a json object
  println(result);
  //make sure the results are not an error
  int status = result.getInt("responseStatus");
  if (status != 200) {
    String reason = result.getString("responseDetails");
    //this sucks that google does not allow you to do very many of these in a row
    println("You are too fast, take a break." + reason);
    return;
  }
  //go one level down in to the results to just get the data not the info about the package
  JSONObject response = result.getJSONObject("responseData");
  //get the arrary of stuff you got back
  JSONArray values = response.getJSONArray("results");

  for (int i = 0; i < min(values.size(),3); i++) {
    //pick out the picture information for each picture
    JSONObject thisGuy = values.getJSONObject(0); 
    String r = thisGuy.getString("url");

    PImage thisImage = loadImage(r);
    permanent = loadImage(r);
    if (thisImage != null) {
      image(thisImage, 770,475, 200, 200);
    }

    //make vOffset bigger so we will draw the next picture below this one.

  }//end for
}

 

Fuel of the Week

I used data from my Nike FuelBand for this weeks project. “Fuel” is a proprietary unit that measures estimated activity over the course of the day. While “3206 Fuel” is meaningless on its own, this unit is an excellent relative measure of activity.

Fuel of the Week collects Fuel data from a .csv file, groups Fuel earned from every day of the week, and performs some math operations to create what we see below:

FergieFuel

Monday = m, Tuesday = t, Wednesday = w, Thursday = h, Friday = f, Saturday = s, Sunday = u.

Pressing a corresponding key will activate that day of the week. The size AND color of the column correspond to what proportion of weekly activity is attributed to that day (on average). Red = LOW, Green = HIGH. It will also show sample Fuel units from a day on that day of the week.

TIL it’s a good rule of thumb to conceptualize the final product before jumping into the code. This code can probably be made more succinct through use of a couple if statements and for loops.

Download the application here. Here is the code:

//Array of data
String [][] csv;
//sums
float mondaySum; float tuesdaySum;float wednesdaySum; float thursdaySum; float fridaySum; float saturdaySum; float sundaySum;
//proportions
float mondayP; float tuesdayP; float wednesdayP; float thursdayP; float fridayP; float saturdayP; float sundayP;
//cumulants
float mondayC; float tuesdayC; float wednesdayC; float thursdayC; float fridayC; float saturdayC; float sundayC;
//colors for mapping
color mon; color tue; color wed; color thu; color fri; color sat; color sun;
//counter and total for proportions
float counter; 
float total;
//visual presets
PFont font;
PImage swoosh;

void setup() {
//the basics
size(1000,1000);
font = loadFont("CenturyGothic-Bold-48.vlw");
textFont(font,32);
swoosh = loadImage("swoosh.png");

//CSV magic i borrowed from the internet
String lines[] = loadStrings("NikeFuel.csv");
int csvWidth=0;
//calculate max width of csv file
for (int i=0; i < lines.length; i++) {
  String [] chars=split(lines[i],',');
  if (chars.length>csvWidth){
    csvWidth=chars.length;
  }
}
//create csv array based on # of rows and columns in csv file
csv = new String [lines.length][csvWidth];
//parse values into 2d array
for (int i=0; i < lines.length; i++) {
  String [] temp = new String [lines.length];
  temp= split(lines[i], ',');
  for (int j=0; j < temp.length; j++){
   csv[i][j]=temp[j];
  }
  }

// make variables: sum, proportion, cumulants, color maps
for (int i=0; i<csv.length-6; i+=6) {
  counter = counter + i;
  mondaySum = mondaySum + parseInt(csv[i][1]);
  tuesdaySum = tuesdaySum + parseInt(csv[i+1][1]);
  wednesdaySum = wednesdaySum + parseInt(csv[i+2][1]);
  thursdaySum = thursdaySum + parseInt(csv[i+3][1]);
  fridaySum = fridaySum + parseInt(csv[i+4][1]);
  saturdaySum = saturdaySum + parseInt(csv[i+5][1]);
  sundaySum = sundaySum + parseInt(csv[i+6][1]);
  }

total = mondaySum + tuesdaySum + wednesdaySum + thursdaySum + fridaySum + saturdaySum + sundaySum;
mondayP = mondaySum/total;
tuesdayP = tuesdaySum/total;
wednesdayP = wednesdaySum/total;
thursdayP = thursdaySum/total;
fridayP = fridaySum/total;
saturdayP = saturdaySum/total;
sundayP = sundaySum/total;

//cumulants
mondayC = width*mondayP;
tuesdayC = mondayC + width*tuesdayP;
wednesdayC = tuesdayC + width*wednesdayP;
thursdayC = wednesdayC + width*thursdayP;
fridayC = thursdayC + width*fridayP;
saturdayC = fridayC + width*saturdayP;
sundayC = saturdayC + width*sundayP;

mon = color(map(mondayP,.13,.16,255,0),map(mondayP,.13,.16,0,255),0,10);
tue = color(map(tuesdayP,.13,.16,255,0),map(tuesdayP,.13,.16,0,255),0,10);
wed = color(map(wednesdayP,.13,.16,255,0),map(wednesdayP,.13,.16,0,255),0,10);
thu = color(map(thursdayP,.13,.16,255,0),map(thursdayP,.13,.16,0,255),0,10);
fri = color(map(fridayP,.13,.16,255,0),map(fridayP,.13,.16,0,255),0,10);
sat = color(map(saturdayP,.13,.16,255,0),map(saturdayP,.13,.16,0,255),0,10);
sun = color(map(sundayP,.13,.16,255,0),map(sundayP,0,.13,.16,255),0,10);

}

void draw () {
    noStroke();
    if (keyPressed) {
    if (key == 'm') {fill(mon);rect(0,0,mondayC,height);
    for (int i=0; i<csv.length-6; i+=6) {fill(255);text((csv[i][1]),0,i*6);};}
    if (key == 't') {fill(tue);rect(mondayC,0,width*tuesdayP,height);
    for (int i=0; i<csv.length-6; i+=6) {fill(255);text((csv[i+1][1]),mondayC,i*6);};}
    if (key == 'w') {fill(wed);rect(tuesdayC,0,width*wednesdayP, height);
    for (int i=0; i<csv.length-6; i+=6) {fill(255);text((csv[i+2][1]),tuesdayC,i*6);};}
    if (key == 'h') {fill(thu);rect(wednesdayC,0,width*thursdayP, height);
    for (int i=0; i<csv.length-6; i+=6) {fill(255);text((csv[i+3][1]),wednesdayC,i*6);};}
    if (key == 'f') {fill(fri);rect(thursdayC,0,width*fridayP,height);
    for (int i=0; i<csv.length-6; i+=6) {fill(255);text((csv[i+4][1]),thursdayC,i*6);};}
    if (key == 's') {fill(sat);rect(fridayC,0,width*saturdayP,height);
    for (int i=0; i<csv.length-6; i+=6) {fill(255);text((csv[i+5][1]),fridayC,i*6);};}
    if (key == 'u') {fill(sun);rect(saturdayC,0,width*sundayP,height);
    for (int i=0; i<csv.length-6; i+=6) {fill(255);text((csv[i+6][1]),saturdayC,i*6);};}
    }
    else {background(swoosh);
    fill(0);
    text("Press & Hold M,T,W,H,F,S, or U",300,.7*width);};
    fill(255);
}