Data Rep » ITPeed

Data Rep: Week 12

Apr 192011

### 1. Gather Data
Data & Participatory Practice
Citizens as sensors: the world of volunteered geography
Volunteer GIS
OpenStreet Map

Participatory Sensing: Build a large human network of sensors
- collecting data for open community
- using mobile phones it can be a geo-tagged data
- Open Data Kit: free participatory sensing tool

Opportunistic Sensing: Data can be accessed whether you like it or not
esssseesbqqchiucuknfnvniovnfn geetha sucks

Seed Drawings
Clement Valla – Draw a line

jammy!!!
geetaaaaaarrrrrr(d)!!!
sleep time
zzzz

### 2. Process Data
System Esthetics – Jack Burnham, 1968
- aim to reduce gap btwn science and artist
- blur boundaries btwn something utilitarian and scientific Yeaaaa!

Data Rep: Week 10 – Sonification

Data Rep No Responses »

Apr 052011

eyes – good for recognizing patterns
ears – good for recognizing time, continuum: good at TEMPORAL (time) RESOLUTION –> SPATIAL RESOLUTION

We have a super accurate temporal resolution in our brains. We can hear auditory spacing with much more distinction than with visual
- eg. Amanda cox: alpine skiing

chordal structure – very good at knowing when two things happen at once. In visual form, it’s much harder to separate, but auditorily? we can do it with ease
- eg. Brain blues

dangers of sonification – easy to make things seem like a pattern when there isn’t. You need to make sure you’re playing something from the data and not just fromt he musical scale
- eg. ToneMatrix

Data Rep: Week 9 – Geospatial Data Techniques

Data Rep No Responses »

Mar 292011

1. Get Lat and Long
2. Find an image you want to display info on top of
3. Make sure points are where u want
4. Plot

Static data – data is always the same
Dynamic data – every time you ask for it it’s different (eg. itp listserv)
Giant dynamic data – Large of amount of dynamic data (eg. itp listserv)

Range of techniques to deal with GIANT DYNAMIC DATA:
Mongo
+ you don’t have to know exactly what your data will look like in the end
+ you can put whatever data you want into it
-

1. Install Mongo
a. get MacPorts
b. sudo port install mongodb
2. Import Data
a. Make a database: type ‘use my Database’ and it will make it for you
b. use mongoimport to import CSV

Mongo –JSON -BSON
Spits out whatever it is you want
even if not all categories have the same tags

{name:”Bob” age:”17″ gender: “M”}
Ask for whatever you want.
If you want height, it’ll get height from categories that have it

mongoimport –host localhost –db foo –collection bar –type csv –file temp.csv –headline

3. Connect Mongo to Processing

Geospatial/temporal programs

TileMill/MapBox
+ relatively easy to create map experiences

Google Fusion Tables
+ Allows you to take 1 or more tables of data, merge them, and create visualization of those things
~ it’ll go through your tables, go through things that look like cities, and assign a lat and long to it
~ http://maps.google.com/maps/geo?output=json&key=abcdefg&q=brooklyn
~ you can change out the stuff in bold
– limited to 20 per day
– if you do it in a spreadsheet, you can have it automatically fill it in the lat/long/confidence for you

Finding LAT/LONG in a GOOGLE SPREADSHEET
Google Doc

Processing – to be emailed over

Aaron Koblin – Flight Patterns
+ Super simple

Eric Fischer – The Geotagger’s World Atlas
~ takes speed of picture taking on a map
~ the black is slower
~ red means you’re moving faster
~ more ppl tag photos not at home, on coasts
~ red = tourists
~ blue = nykers
~ makes something from nothing

Pedro Cruz – Lisbon’s Blood Vessels
~ traffic data
~ many iterations

Ways to Map out GEOSPATIAL DATA
{LatLon} –>Projection Formula–> Cartesian Coordinates

Mercator Projection (Gerardus Mercator)
link
- Won’t work because it’s not equal grid
- But you can run it through a mercator formula and it’ll put it out properly

Winkel Tripel (Oswald Winkel)
link

Dymaxian Projection(Buckminster Fuller)
link

Sphere
- Don’t deal with a projection at all, just plot everything on a SPHERE in processing
- how do i turn lat and long into spherical coordinates and vice versa????
- half of it is hidden at any given time (issue) which makes data not as easy to look at at once
- only time you’d want to use a sphere is when you want to show continuity on globe (no “end of the map”)
Good morning – Jer
- 3 layers. Map, City lights, clouds

1. Use known marker points
2. Tweak origin, x/y scale to match
3. 0,0 (off coast of Africa)
Hawaii
Tokyo
London

Next week. Sound and Data

Data Rep: Class 8 – Interactive Data Viz

Data Rep No Responses »

Mar 232011

Golan Levin – The secret lives of numbers
- example of how even the most trivial data set can be interesting
Picture
Check it out

Steve Varga – Pennant
- Design heavy exploratory tool (clickable nodes to go into things) (possibly the future of data viz)
- Able to go into granular detail with the data. Eg: Pick a date, find a team, find a player, how many home runs that player made.
Video

Fernanda Viega – PostHistory
- Had a viz lab at google
- allows them to visualize their email database

Moritz Stefaner – Relational Browser
- Relational data set Eg. Ppl in switzerland speek swiss, what other countries speak swiss
- Demo

Fizz – Bloom.io
- a company that creates exploratory tools
- Most exploratory data is not share-able
- This visualizer twitter feeds
- critique: good feel/physics, but no real thematic clustering
– it’d be nice with a time slider to see when topics burst
– the way these clusters are arranged doesn’t tell a story about what you’re showing
— if you compare two clusters what differences/similarities can you see? (as of now you can’t tell)
link

Mitchell Whitelaw – commonsExplorer
- it allows you explore the database of photos on flicker that are creative commons
link

Karsten Schmidt – Social Collider
- Visualizes twitter as a particle acceleration
- wrote the toxiclib library
link
- we can see ppl are very “bursty” with tweets and that’s true with human behavior in general
thought: data viz game? A game stage set by data??

Question: Is there code that can be applied across many sets of data
- pros: good for things like linguistics, networks, yes. Also very useful to have it apply to many things
- cons: if you want to get specific data from each of these data sets, then you lose that granularity

Matthew Bloch, Shan Carter & Alan McLean – Mapping America (NYTimes)
- asking tons upon tons of people to look at a single data set and see what stories come out of it
– asks to share on facebook
link

Gary Blight & Sheila Pulham – Arab Spring (The Guardian)
- interactive exploratory visualization
link

We have tools that allow us to:
change focus
shift perspective
follow connection (twitter trail guy)
leverage collaboration

exploration
- manual
- automatic
- assisted – here are your results, here are some related results that can drive you down other avenues

Data
- forms a landscape
- fitness: best idea is most fit
- big jumps, then smaller jumps to find the best places

make

ideas: data rep as game
more interactive interesting news feeds

Data Rep: Class 7

Data Rep No Responses »

Mar 082011

www.mongodb.org

bit.ly.itpdemo1

Spring Break Homework:
Produce a dataset
- more than 200 datapoints stored in a Google Spreadsheet everyone must contribute equally
- eg. feltron report stuff: distance traveled, sleeping hours, number of drinks consumed
- what kind of question would you want to answer with this data?
- check out datum
- download iphone app

Group:
andi, dave, marko, myself

code – update key string
input

hex:
unhex:
When inputting color: FFFF9900 (ARGB)
(alpha ff)

while loop – easy to get stuck in an infinite loop

**double check the sketch for accurate code:
group = 4 || sc = 16 (something like that… if group = 4 unless last group then 16 is OK.

Data Rep: Week 6

Data Rep No Responses »

Mar 012011

NYTimes Article Search API: www.blpnt.com/source/nytimes.zip
1. Times engine –> 2. TimesArticleSearch –> 3.NYTArticleSearch API –> 4. TimesArticleSearchResult –[ 5. times results object/timesfacetobject

2. TimesArticleSearch
addQueries("search,terms")
addFacets("facet1,facet2") <- indicate what facets you want back (eg. geo_facet/ per_facet/ org_facet/ des_facet etc)
addExtra("extra","value") <- could add a begin/end date. listed on developer side of api. Not used too often
addFacetQuery("facet","value") <- use facet as query. (eg. if i only want publications for the month of feb, put in "month_pub" for "facet" and value "1?" not sure why.

2-4. doSearch()

4. TimesArticleSearchResult
int total (what's the total number from my search?)
[TimesResultsObject] results (contains object results/actual results)
getFacetList(“facet”) –> [TunesFacetObject] (contains facet results)

5. TimesResultObject: these are the things that’ll be inside. You can filter for whatever you actually want.
_abstract
body
byline
date
lead_paragraph
title
url

5b. TimesFacetObject
count (how many times counted)
term (what is the term: eg. Libya etc.)

______

import blprnt.nytimes.*;
String aKey ="ebb930bb67d57dfdd2892a6c5af4c51b:19:54875063";

void setup() {
size(700,500);
background(255);
smooth();

TimesEngine.init(this, aKey); //initialize and reference 'this' application
TimesArticleSearch mySearch = new TimesArticleSearch();
mySearch.addFacetQuery("geo_facet","RWANDA");
mySearch.addFacetQuery("publication_year","1994");
mySearch.addFacets("publication_month");
TimesArticleSearchResult r3 = mySearch.doSearch();

TimesFacetObject[] months =r3.getFacetList("publication_month");
println("Rwanda:" + r3.total);
println("January results:" + months[0].count);

for (int i=0; i<12; i++) {
fill(random(150,255),0,0);
float w =width/12;
rect(int(months[i].term) * w,height,w,-months[i].count*3);
}
}

void draw() {
}

getFacet = won’t work if you haven’t done addFacet
geo_facet = terms you get out are facet NAMES not # unlike with publication_month

Data Rep: Week 5 – JSON

Data Rep No Responses »

Feb 222011

Propositional density = deep proposition/surface proportions
The propositional density of an object is the number of deep propositions divided by the number of surface propositions it conveys.

Propositional density is the amount of information conveyed by an element or design per unit element.
surface propositions (Ps) – conspicuous and perceptible. The rectangle is blue
deeper propositions (Pd) – underlying and often hidden. The blue rectangle conveys calm, trust, stability, sadness, etc

This image causes a bit of controversy between technical and artistic visualization. This mountain/stock market makes the data more engaging, but also makes it less technical.
Technical vis <-------------------> artistic vis
easy to read harder to understand quickly
boring interesting to look at

###JSON: JavaScript Object Notation

Data storage:
{
Curly Braces – OBJECT //as an object stucture
}

[ Square Brackets - ARRAY] //as a list

example:
{ //Information stored as OBJECT
name:”Peter”, //STRING
age:18, //INT?
smoker:false, //BOOLEAN
friends:["April", "Ron","Valerie"] //ARRAY (objects and arrays can be stored within each other eg. each of these friends could be an object with smoker values too. It’d be an object in an array in an object)
};

How to access stuff in JSON: (Super easy)
myJSONObject.age = access age “18″
myJSONObject.friends[1] = access “Ron”

When accessing things from JSON in JAVA: it’s not easy
myJSONObject.getInt(“age”);

3. try{
4. String url — etc.
4. String string Data — etc.
2. myJSONObject = new JSONObject(stringData);
1. JSONArray a = myJSONObject.getJSONArray(“friends”);
println(a.getString(1));
} catch (Exception e) {println(“JSON LOAD FAILED.”);
}

It needs all this to help it run and double check incase it fails to load etc.

//Note to self: add JSON library to processing

api:

http://developer.nytimes.com/docs/article_search_api

To search api on a page:
?query=libya&api-key=#####

so in processing:
void getJSON() {
String keywor = “libya”;
String url = endPoint + “querty=” + keyword + “&api-key =” + apiKey;
}

99.9% of the time we’ll use double quotes
” ” – stings
‘ ‘ – characters (single characters: used for things like KeyPressed ‘a’ etc)

http://api.nytimes.com/svc/search/v1/article?query=libya&api-key=ebb930bb67d57dfdd2892a6c5af4c51b:19:54875063

Get a looong string of JSON:
{“offset” : “0″ , “results” : [{ "body" : blhablhablahbalhbalbha} , {same thing again}], “tokens” : ["libya"] , “total” : 8820}

http://prototype.nytimes.com/gst/apitool/index.html

Allows you to filter for data you’re looking for.

Facets: data surrounding your term (in this case libya)
per_facet // people per term
geo_facet //geological data
org_facet //organization related to data, (UN, etc.)
des_facet //descriptions related to, middle eat conflict
publication_year //pub year
publication_month //publication month

http://api.nytimes.com/svc/search/v1/article?query=(field:)keywords (facet:[value])(&params)&api-key=your-API-key

To search for facet type per search
void getJSON(String keyword, String facet, String facetType) { //added facet & faceType: ?query=libya_facetType:facet&api-key=####
//String keyword = “libya”;
String url = endPoint + “query=” + keyword + “+” facetType + “:” + facet + “&api-key=” + apiKey;

PI/15

Data Rep: Class 3 – ArrayLists

Data Rep No Responses »

Feb 082011

~transfer/EgyptCount.zip

1. import (CSV, JSON, XML)
2. parse **
3. display (android, sound, print, computer, etc.)

** A very central part to data rep. A lot of freedom @ 1 & 3.

Parse —-> Data Object
input: from XML()
from JSON()
output: render

Arrays
+ fast
+ easy to use
+ good for known qualities
- bad for unknown qualities

ArrayLists
+ flexible
+ full-featured
+ good for changing quantities

Basic ArrayLists:
ArrayList myList = new ArrayList();
CheeseBurger cb = new CheeseBurger();
myList.add(cb);

Ideally: DOES NOT WORK
CheeseBurger cb = myList.get(0);
cb.eat();

Really:
CheeseBurger cb = (CheeseBurger) myList.get(0);
cb.eat();

What works (in processing?):
ArrayList myList = new ArrayList();
CheeseBurger cb = new CheeseBurger();
myList.add(cb);

the “< >” is a generic that tells the ArrayList that it’s a “CheeseBurger.”
Only capital letters go in here.

Things that can be put into an array list:

String
PImage
(anything with aCAPITAL letter. It’s a real object)

anything like:
int
float
you need java to help.

You can use “Float” or “Integer” they hold the place of int and float.

###HashMaps (similar to the way we remember things. By keys rather than numerical)
o store objects by a key (almost always a String)
+ helpful for keeping counts of categories
+ useful for keeping checklists

How it works:

HashMap
myMap = new HashMap();

[put in hashmap]
CustomObject o = new CustomObject ();
myMap.put(key, o);

[get using key]
CustomObject o = myMap.get(key)

Example (Processing)

XML
A query is always name=value&name=value etc.

Object
o describe it
o what does it do?

/*
We Feel Fine Visualization
jamietlin@gmail.com
Feb 8, 2011
1. go to api
2. pull an xml document
3. get the data in. Send to xml element (takes text file and makes it useful for us) (XMLElement)
4. Get those 500 children (getChild) but be able to filter for the children you want
5. Run the not null feeling and populate/fill arraylist from the XML
6. Throw these words on a random place on the screen
--NEXT STEP: find which feelings are popular now by categorizing
*/

//1. both base & query required
String baseURL = "http://api.wefeelfine.org:8080/ShowFeelings?";
String query = "returnfields=feeling,sentence&display=xml";
ArrayList<FeelingEntry> feelingEntries;

void setup() {
size(750,750);
background(0);
smooth();
getFeelings();
renderFeelings();
}

void draw() {
}

//functions made down here to keep setup & draw clean
void getFeelings() {
//2. Make the feeling list to put these things below into it
feelingEntries = new ArrayList();
//3. Import XML into XMLElement
XMLElement xml = new XMLElement(this, baseURL + query); //'this' is a reference to the app
println( xml.getChildCount() ); //returns the # of <feeling> nodes inside the <feelings> node (parent child in XML)

//For every child node, make a feeling entry and populate it from the XML
//pseudo code below and slowly fill in what you know:
// for (every child node) {
// make a feeling entry;
// populate it from the child node;
// }
//4. first initialize it, tell it how far to run, tell it how to run
for (int i = 0; i < xml.getChildCount(); i++) {
FeelingEntry f = new FeelingEntry();
//5. run fromXML method. Which takes a word from feeling or sentence in the "FeelingEntry class"
f.fromXML( xml.getChild(i) );
//if our new FeelingEntry has a valid feeling property, add it to the list
if (f.feeling !=null) feelingEntries.add(f);
}

println( feelingEntries.size() );
}
//6. rendering
void renderFeelings(){

//pseudo code: Go through each feeling and show it at a random place
// for(each feeling) {
// show the feeling at a random place;
// for(int io;i <feelingEntries.size(); i++ {
//etc.
//}
for(FeelingEntry f:feelingEntries){
//put these words on a random position on screen
pushMatrix();
translate(random(width), random(height));
//make a text field for the FeelingEntry's feeling
fill(255);
//println(f); //test to se if f or feeling below is null (when you get NullPointerException)
text(f.feeling, 0,0);
popMatrix();
}
}

ITPeed

Data Rep: Week 12

Data Rep: Week 10 – Sonification

Data Rep: Week 9 – Geospatial Data Techniques

Data Rep: Class 8 – Interactive Data Viz

Data Rep: Class 7

Data Rep: Week 6

Data Rep: Week 5 – JSON

Data Rep: Class 3 – ArrayLists

Categories

Archives

Links

Meta