|
Syllabus / Visualizing-the-Five-Senses-F08
Visualizing the Five SensesJane NisselsonTuesdays 9:30am-12:00NSYLLABUS. VISUALIZING THE FIVE SENSES. H79.2720.1 INSTRUCTOR: JANE NISSELSON. September 6/08 OVERVIEW This syllabus will be updated weekly during the semester with current assignments, readings, viewings and references. During the term, we visualize and sketch (acquire, parse, filter, mine, represent, refine, and interact) sense data (sight, sound, touch, smell, taste) while looking at the role of visualization in science. There will be readings, movies, video clips, web references, and speakers on these three topics: visualizing data, scientific investigations, and mapping our senses.
Grading You start with an A. To keep it: complete projects on time, attend class on time, and participate. Read assigned articles. We will not have time during the class to discuss every reading but they are essential to the class and should inform your sketches. If you do not consistently attend on time, do not expect to pass; forewarned, take afternoon and evening classes.
Assignments The class has bi-weekly assignments using the Processing environment to present 5 data sets. Three quarters of the term is dedicated to sketching the five senses. These exercises are a prelude to a final project. Proportionally, your grade is based 75% on the five sketches and 25% on the final project. In the immortal words of Ben Fry, one of the creators of Processing, "Don't start by trying to build a cathedral." Our goal is to use code to test ideas and hypotheses with ease and create meaningful visuals. The term starts by being playful with the senses -- visualizing something you know; the second half focuses on real science problems.
URL The current syllabus and links to class assignments and contentL http://itp.nyu.edu/v5s/
The class varwiki and current syllabus: itp.nyu.edu/varwiki/Syllabus/Visualizing-the-Five-Senses-F08
CLASSES AND ASSIGNMENTS CLASS 1: 09/02: SOUND What sense is your favorite; which could you live without?
Students backgrounds and goals.
Assignment 1A Revisualizing sound: Building a meaningful equalizer from minim. You can use the sample code below as a starting point. Sound has been visualized in a number of ways from waves to bad ipod equalizers. How would you know it was sound? How would you explain it to someone who's deaf. What is purpose of visualizing sound differently? Could you think of it as drawing with your voice?
minim audio library for processing. http://code.compartmental.net/tools/minim/
sample code: http://itp.nyu.edu/v5s/CLASS01/class01.html SPECTOGRAM.zip (author Carl Tashian)
Assignment 1B Look for representations of that sense data that appeal to you; bring in one. Be ready to say what is its appeal is to you. What’s the lead medium: illustration, modeling, photography . . .
Reference Visualizations AEG sound level billboard. http://www.noiseawareness.blogspot.com/
Test Your Hearing. http://www.phys.unsw.edu.au/jw/hearing.html
Reading Truxal. Chapter 4: The building Blocks of Signals
Mixed Feelings By Sunny Bains. http://www.wired.com/wired/archive/15.04/esp.html
"Visualizing Data". Ben Fry. Chapter 1. The Seven Stages of Visualizing Data.
Viewing "Powers of Ten." Charles and Ray Eames. 1968.
"In My Language". The native language of autism. http://www.youtube.com/watch?v=JnylM1hI2jc
CLASS 2: 09/09: SOUND Building on your assignment: equate with human bandwidth; harvest audio data; interactively "exploding" a sample; perception filtering and mapping sounds. Use the tools in the minim library to represent your sounds. You can use the sample music: pink.mp3, create your own sample, work from values in any of the reading or references.
References "The nature of the Soundwave" http://www.glenbrook.k12.il.us/GBSSCI/PHYS/CLASS/sound/u11l2b.html
minim library FFT example. http://code.compartmental.net/tools/minim/manual-fft/
http://itp.nyu.edu/v5s/CLASS02/class02.html -- copy of powerpoint lecture
Reading Truxal. Chapter 6. An Engineer Looks At Hearing.
"Visualizing Data". Ben Fry. Chapter 2. Getting Started with Processing.
"HELLO, HAL Will we ever get a computer we can really talk to?" John Seabrook. The New Yorker. 06.23.08
"Making the visible invisible." Stuart McKee. Eye 57 p 68-74.
Viewing The Conversation. Francis Ford Coppola. 1974
Film Ist. [1-12] Gustav Deutsch. Index Films
CLASS 3: 09/16: SIGHT Sampling from microscope. Sampling from microscope. Create a written description of the visualization you are going to create and the quantification strategy you are going to use. First list what idea you want to explain or demonstrate. What's the difference between what we see with our natural eye, 20X, 40X magnification? What's changing, being lost, gained when you translate the image. One project could be simply to compare different printed pages' dpi and print qualities.
Where are you going to direct the viewer's attention? How does your visualization make your idea accessible? What are you going to measure? What are you going to filter? Does your visualization take into account: framing, color, labels? Can you connect another sense to the image such as sound? How can sound relate to vision: can we recognize a frequency? We want to define what our aims are in the visualization before we start programming.
Check out the QX3 microscope from the equipment room and download the libraries to use it (see below). Does the resolution / brightness etc change your strategy for what you want to look at and visualize? Can you define a visualization problem that is relevant to this? Look at the work of artist Jim Campbell to see how his work challenges our understanding of resolution and vision (and sound!); particularly his "ambiguous icons"
Using the QX3 Install the macam library to enable quicktime to talk to the microscope. download from link or download from CLASS03; sample software is also there. You need to copy the macam.component to the quicktime folder in your system's library folder. Next, install the processing video library for routines to control the video data from camera. The list routine has an example of a sketch that will list the devices. The capture routine lets you talk to the microscope. The QX3 device is "Intel Play QX3 Microscope". There is sample software at the CLASS03 link on http://itp.nyu.edu/v5s. There are 2 microscopes in the equipment room to check out; a third should be there by friday, end of day.
Assignment Links celestron microscope driver http://www.celestron.com/c2/product.php?CatID=31&ProdID=45
macam : USB webcam library http://webcam-osx.sourceforge.net/
processing video library: http://processing.org/reference/libraries/video
list video devices (use example) http://processing.org/reference/libraries/video/Capture_list_.html
capture video from microscope http://processing.org/reference/libraries/video/Capture.html
sample software CLASS03 link on http://itp.nyu.edu/v5s
Reading "The End of Theory: The Data Deluge Makes the Scientific Method Obsolete" Chris Anderson. 06.23.08
Viewing Blow-Up. Michaelangelo Antonioni. 1966.
The Conversation. Francis Ford Coppola. 1974
References Work of artist Jim Campbell http://jimcampbell.tv/
Nikon | Feel Nikon | Universcale http://www.nikon.com/about/feelnikon/universcale/index.htm
Image and Meaning http://www.imageandmeaning.org/index.html
Apple. Science. Profiles. Felice Frankel http://www.apple.com/science/profiles/frankel/
Apple. Science. Inside the Image. B-Z http://www.apple.com/science/insidetheimage/bzreaction/
powerpoint lecture: vision http://itp.nyu.edu/v5s/CLASS03
Guest Speaker George Prochnik 11am. Reading from his upcoming book on sound.
CLASS 4: 09/23: SIGHT Sampling from microscope part 2
Enhancement to Vision/Microscope assignment: What is 1X?
Take the problem you've created and add a scale measurement to it. How do you show comparative scale; what does it mean to be 1x, 20x, 400x. Come up with a method to measure scale of data; possibilities (A) a real ruler under scope (B) compare / include to something whose size we know; (C) scale bar inside or outside the image; (D) expands scale as you zoom in; (E) can be a time series in small multiples with measure; (F) can start with digital camera; (G) consider adding pixel interpolation between data using the integrator class (Visualizing Data); (F) Possible objects to look at: natural: eggshells, leaves, sand, sugar, hair, fingers; manmade: microchips, money notes, coin, magazine paper, newspaper (dpi)
A visualization is about comparing.
Why add processing? Showing more than you can in one screen. Creating user interface to static data.
Reading Tufte. (emailed ET.pdf) Visual Explanations. Graphics Press. 1997. p 13-25.
Truxal. Chapter 11. p 418. Appendix 11.1 Human Vision
Truxal. Chapter 8. Signals through Space p 261-268
"Envisioning Science." Felice Frankel. MIT Press. 2002 ISBN 0-262-06225-9. Chap 3. Images in Science: A Gallery of the Past. Phylis Morrison. p14-23.
"On the Surface of Things: Images of the Extraordinary in Science." Felice Frankel. Harvard University Press. 2008. ISBN-10: 0674026888.
"Insights of Genius." Arthur I. Miller MIT Press. 2000. ISBN 0-262-63199-7. Chapter 8. Visual Imagery in Scientific Thought. p263-324.
"The End of Theory: The Data Deluge Makes the Scientific Method Obsolete" Chris Anderson. 06.23.08
http://www.wired.com/science/discoveries/magazine/16-07/pb_theory References Visualizing Data. Ben Fry. Integrator class: p 48-51.
amazing visual topics http://www.edwardtufte.com/bboard/q-and-a?topic_id=1
Microscope Drawing Workshop http://www.flickr.com/photos/scalefreenetwork/
Viewing Blow-Up. Michaelangelo Antonioni. 1966.
Fight Club. David Fincher.
CLASS 5: 09/30: SMELL 01 PART I Preparation: "Bilateral olfactory sensory input enhances chemotaxis behavior": (from dr. vosshall's research that she discussed in class): review powerpoint: http://itp.nyu.edu/v5s/CLASS05/ASSIGNMENT%2005/bilateral.ppt
Exercise: For this assignment, we are not programming. You have the opportunity to create models for the representation of the data Dr. Vosshalll's research group collected on the navigation of genetically modified drosophila larvae through an odor field.
This exercise focuses on the creative and conceptual pre-process that takes place in designing a visualization. Before programming, you need to design the elements that will appear in the visualization. We refer to this as the "modeling phase." The design of the elements should enhance and drive the concept behind the visualization.
Your assignment is to do image research for representing the components of the visualization (listed below). These should be inspirations for how these components will be modeled later on the computer. They can range from excruciatingly scientifically detailed to provocative interpretations. Think about the sensuousness of: smell, scent, being enveloped in scent, searching for something by tracking the intensity of the smell, etc. If you can't find images that adequately represent these components, then you can create them in any media.
1. 2 paths (unilateral and bilateral olfactory)
2. odor (particle and odorant gradients)
3. larvae
4. define space of experiment (2d / 3d / background / graphed etc)
Materials can be photos, illustrator files, drawings paintings, images from books or magazines, parts of other videos, photos that you take, etc. Try something new, be adventurous, think about how your models can really telegraph the conclusions of the research.
Keep in mind, your presentation of the elements is part of the assignment (show as a pdf / powerpoint / physical objects / collage/ blog entry / photo album).
At this point, the 4 elements which you are designing do not have to be assembled in the visualization; we will move into that in the following week. You want to spend time finding inspirations for the elements you are modeling and also zeroing in on what is important to communicate in the visualization.
PART II Post a link in your blogs to 1. one visualization you like that effectively communicates its subject matter – it can be about anything.
2. one visualization that is about a sense (vision, hearing, taste, smell, or touch)
Analyze how the visualization communicates its subject matter. Analyze the particular components used to drive the visualization. This editing process is key to making an intelligent visualization. Reading http://itp.nyu.edu/v5s/CLASS05/ASSIGNMENT%2005/LouisVosshallNN07.pdf "Bilateral olfactory sensory input enhances chemotaxis behavior." Nature Magazine. Matthieu Louis1, Thomas Huber2, Richard Benton1,3, Thomas P Sakmar2 & Leslie B Vosshall1
"Visualizing Data". Ben Fry. Integrator class: p 48-51; interpolation between data sets p 87-92
References powerpoint lecture: vision & smell http://itp.nyu.edu/v5s/CLASS05/VISUALIZE.ppt
Guest Speaker: Leslie Vosshall. Head of the Laboratory of Neurogenetics and Behavior at Rockefeller University and a Howard Hughes Medical Institute investigator
CLASS 6: 10/07: SMELL 02 PART 1 INDIVIDUAL "Visualizing Data". Ben Fry. Chapter 6. Scatterplot Maps. p 145 - 181
Please complete the entire visualization, working step by step, through the exercise. I am assigning this chapter for a number of reasons. From a programming point of view, it covers setting up, reading, and drawing a database. From a design point of view, it demonstrates ways to visually prioritize aspects of the data. If this is easy for you, great, you can complete it in just an hour or so. If it's hard then this will be a very rewarding exercise which will walk you through some basic skills used in making a visualization.
Another reason I am assigning this is one of the life skills I took away from MIT is the discipline of sitting down, reading, and following instructions in order to learn how to do something. Building the discipline to follow instructions and incrementally create something (from a manual, tutorial, or reference website) will give you a huge amount of independence. You learn how to work methodically and problem solve on your own. Also, any project for a client (whether an art gallery, a commercial enterprise, or a scientist) requires a time commitment and incremental work (often from a team) in order to transform it into something that has significant visual impact.
Please post a link in the varwicki to your final visualization.
PART 2 ODOR GRADIENT For the project on visualizing the Vosshall database
http://itp.nyu.edu/v5s/CLASS05/: ITP SMELL.zip program for use in eclipse
Reading / Exercise "Visualizing Data". Ben Fry. Chapter 6. Scatterplot Maps. p 145 - 181
10/14: [NO CLASS SCHEDULED]: SMELL 03 Assignment 6 contd For the project on visualizing the Vosshall database
http://itp.nyu.edu/v5s/CLASS06/: smell3.zip program for use in processing
http://itp.nyu.edu/v5s/CLASS06/: DataLoading.zip simple routines for loading xtml path and odor gradient files (if you aren't using the full programs.
CLASS 7: 10/21: SMELL 04 Group Work on Smell Assignment.
Refining the visual quality and communication impact of the Vosshall visualization.
1. Odor (gradient): a. intensity (how strong odor is); data more or less particles on grid location; b. characteristics – quantity; what it looks like; c. odor source. What is it?
2. Nose / larvae (maggot) – doing detection: a. Entity’s appearance; b. Detection? Navigation; c. 2 kinds (bilateral + unilateral
3. Path / trajectory / time * space: a. persistence; b. datanoid. Nonabstract – slime, footprints, planetrail – (color, weight, persistence…) what it looks like should convey information (velocity .. type; show relation in successive paths; c. why it is deviating; e. time: sampled 6x per second
4. legend: enhance reading
5. bounding box
Reading (For Next Week) http://www.tastescience.com/abouttaste1.html Cornell University Taste Science Laboratory
http://benfry.com/writing/archives/4 "Restoring Sight with the Tongue"
http://www.newyorker.com/reporting/2008/05/12/080512fa_fact_max "A Man of Taste: A chef with cancer fights to save his tongue" by D.T. Max. May 12, 2008.
CLASS 8 10/28: TASTE The program provided in this week's assignment creates a data network from wine notes. For this assignment I would like you to use processing to create a network representation for the database that is different from the one provided in the example which is organized around a circle. What other layout and connections can you create for the database? Like the smell visualization, we have a set of elements to work with: database, sort mechanism, nodes, and links.
Notice that the assignment is about organizing words. Taste is a subjective sensation so one way to quantify it is through words. The wine community has evolved a vocabulary to parse and classify a huge variety of taste sensations.
Approach this assignment as not simply organizing words. Instead, it is a way of diagramming subjective evaluations to a given stimulus. How can we organize data generated in response to a sensation?
Remember, a great visualization often makes connections that have not been seen before!
All assignments are to be completed in processing or the java eclipse platform.
Options 1. Create or find another database. Feel free to create your own database of network words, food/beverages, and descriptions. You could create one out of a diary of foods you eat in one day; chocolate, or coffees for instance.
2. Add another field to the database that provides another criteria for organizing the database such as an attraction/repulsion factor.
3. Use images instead of words. You could also create a database substituting words with tagged images
4. There is an article in the New York Times about the Molecular Cooking movement. There are a number of links in it to "science" projects which include food; these may pique your interest and provide a source for a database.
Assignment Links To get this running, you may need to install the csv reader jar into your Processing libraries folder, not the sketch libraries folder. Note the csv reader looks in your (on the mac) documents/Processing/ folder for program and its data files.
DATABASE LINKS The tasting notes are from Winestate magazine, from their back issues 1996-2000: http://www.winestate.com.au/
Exported from this program called WineBanq: http://www.winebanq.com
http://www.thenibble.com/reviews/main/wine/aroma-chart.asp Descriptor words merged into broader categories
University of Florida citrus Research and Education Center sorted by Aroma Thresholds -- also has Taste thresholds
http://www.littlefatwino.com/winedb.html a completely visually chaotic wine database
Reading "Design and the Elastic Mind." Paola Antonelli. MOMA. ISBN: 0870707329. "Critical Visualization." Peter Hall. p 122- 130.
References http://www.wired.com/special_multimedia/2008/ff_futurefood_1611 "The future of food"
http://www.gagneint.com/Final%20site/Animation/Pixar/Ratatouille.htm How taste can be visualized in an abstract way.
http://www.foodpairing.be/cooked_chicken.htm Tree graphs of compatible foods
http://www.foodfordesign.be/ Obsessive visualizers from the molecular gastronomy world
http://www.nytimes.com/2008/10/29/dining/29hungry.html?_r=1&ref=dining&oref=slogin Dry-Ice Martini and Electric Cake by Julia Moskin.
THIS WEEK Please be sure to go surely one of the largest sense visualizations at Madison Square Park: http://nymag.com/arts/art/features/51352/
"When you walk up to Rafael Lozano-Hemmer’s installation Pulse Park, which will be switched on Friday evening in Madison Square Park, you register at a kiosk—but instead of giving your name, you record your heartbeat. Two hundred individuals’ pulses then become 200 beams of light, forming a blinking, flashing scrim that parkgoers can walk through."
CLASS 9: 11/04: TASTE 2 Continue refining taste assignment
Lecture Refining your visuals: Typography including artists: Lawrence Weiner, Ed Ruscha, Richard Prince, Josef Albers; designers: M&Co, Paul Rand, George Lois.
Link to presentation in pdf form http://itp.nyu.edu/v5s/CLASS09/CLASS 09 TOUCH.pdf Viewing Touch. The Forgotten Sense. Directed by Kun Chang.
Reading The Itch. Atul Gawande. June 30, 2008 http://www.newyorker.com/reporting/2008/06/30/080630fa_fact_gawande
CLASS 10 11/11: TOUCH Fingerprint assignment. in class. Customize the toucharray.pde program to be your fingerprint. It should reflect you and your sense of touch; additionally can reflect what it is touching. 30 minutes.
Links Fingerprint
http://itp.nyu.edu/v5s/CLASS10 (touchArray folder in the PROGRAM directory)
Assignment FP Begin final project
1. Bring in the question that you want to answer in your final visualization. You can build on one of the assignments we've had so far. Or, tie in with another course you are taking, especially if you are using hardware. Think about the ecology of the entire sense system, how the senses relate to each other and are quantified.
2. Bring in 5 images showing the sense that you want to work on for your final project. the images must be something we can post on the wall -- so find magazine / book images / xeroxes / print outs etc. it can show the sense detector (in the body); and the object (database) it is sensing
3. Montage together your assignments for each of the 5 senses around an image of a human body / head ... and relate each image to the sense detector -- you can do this in photoshop.... so, basically, you need to take a still image from each of the assignments and collage it into one image with the addition of the senses they relate to..... Think about if there is something in common between all the databases -- range of frequencies? variation of intensity? vocabulary?
CLASS 11 11/18: FINAL PROJECT Continue working on final project
Guest Speaker: 9.30 am. Ken Kirshchenbaum Kent Kirschenbaum is a professor at NYU's Department of Chemistry and also a founding member of the molecular cooking group "Experimental Cuisine Collective". He will be showing how taste can be used as a tool in demonstrating topics in polymer chemistry and ion exchange reactions. He will have the students make Mango Caviar, using recipes adopted from the famous restaurant El Bulli in Spain and also speak briefly about chewy ice cream.
Links Kent Kirschenbaum http://www.nyu.edu/projects/kirshenbaum/
the experimental cuisine collective. http://experimentalcuisine.googlepages.com/
CLASS 12 11/25: FINAL PROJECT First presentation of all final projects. Initial programming should be complete.
Reading / Handouts / Discussion Lecture: A response to “The end of theory” Chris Anderson (see class 5)
Responses from the edge. http://www.edge.org/discourse/the_end_of_theory.html
The value of the traditional scientific method: hypothesis, model, tests. (W. Daniel Hillis, Douglas Rushkoff, Kevin Kelly)
Kevin Kelly on petascale computing and correlative analytics. http://www.kk.org/thetechnium/
"The Senses Considered as Perceptual Systems." James J. Gibson / Cornell University. 1964.
Topic: The senses as an active interrelated systems. Handout: Table 1: The Perceptual Systems. gibson.pdf
"Proust Was A Neuroscientist." Jonah Lehrer. exerpt p. 140-143. The corticofugal system. proust.pdf
CLASS 13 12/02: FIRST PREVIEW Full visualization complete and ready for refinements. Each student’s presentation should include an introductory slide with the project name, the question posed, the designer’s name, the sense in question, and credited data sources.
Each student will give a 5 minute presentation of their final project as a work in progress. At the end of the hour, we will critique the presentations as well as the projects.
Initial class project review by ITP professor, Gretchen Gano. Biography. Gretchen Gano is the Librarian for Public Administration and Government Information at New York University Libraries. She is part of a data curation group that is exploring how libraries can support and preserve research data collections across the sciences. She holds a Masters in Library Science and a Masters in Public Policy with a concentration in science and technology policy. She teaches the “Art/Science Collisions: Communicating with Data” at ITP. Ms. Gano previously was Creative Director for the planetarium space show at the Rose Center for Earth and Space at the American Museum of Natural History, where she also designed and produced information graphics, interactive kiosk, and web content about current earth science and astrophysics research.
Viewing Fast, Cheap & Out of Control. Errol Morris. Rodney Brooks segments. Augmenting the senses; robots.
CLASS 14 12/09: FINAL REVIEW Allyson Torrisi, Photo Editor, Popular Mechanics. Paola Antonelli, Senior Curator Architeture and Design, MOMA. Natalie Jeremijenko, artist and engineer, Associate Professor at NYU in the Visual Art Department.
|