Final Project Physical Computing/ICM
Grid Mate.  In collaboration with Ryan Viglizzo

We developed a framework in Processing and Arduino that allows artists, educators, and musicians to attach visuals and sounds (music or voice) to any physical space via sensors. The core idea is simple–place a sensor on a physical space (wall, ground, desk, etc.); create a visual of your choice, upload that visual to processing, project the processing sketch canvas on that space and attach sounds to your sensors. As you press the sensors your visual moves to the sensor that you pressed as well as triggers a sound that can be looped or activated once. There are four aspects to this project: “sensors”, “visuals”, “sounds” and ‘space’. Anyone of these contexts can be modified. But here is how we presented it….

“Sensor”: We picked an FSR (pressure sensor) because of the accuracy of its reading.

“Visuals”: We decided to have a dissolving ball be our visual. So as it moves to the different locations on the canvas it begins to paint the canvas.

“Sounds”: We mixed various samples to have them looped

“Spaces”: We decided to make a canvas be our medium.

—We also incorporated a whiteout function that deleted the entire visual starting it over from scratch. —

In developing this code we realized how malleable it was. Anyone of these parameters can be interchanged. For instance, in the context of musicians you could put sensors on your performance backdrop wall behind you that triggered sounds and visuals to where they touched on the wall. Educators could use this system to attach sensors to a map of the USA where kids could press on states and trigger voice descriptions of the places. The possibilities are endless.

We made this project open source. Anyone can use this code (the sensors are up to you). Please feel free to contact either Veronika or Myself for questions or project ideas using this technology.

Arduino Code: The Arduino code was a simple analog read of the sensors. We made this the most simplistic portion of the code in order to make the communication between processing and Arudino smooth.
void setup() {
Serial.begin(9600);

}
void loop() {
int x = analogRead (A0);       // read the pot value
Serial.print(x);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x1 = analogRead (A1);
Serial.print(x1);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x2 = analogRead (A2);
Serial.print(x2);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x3 = analogRead (A3);
Serial.print(x3);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x4 = analogRead (A4);
Serial.print(x4);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x5 = analogRead (A5);
Serial.print(x5);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x6 = analogRead (A6);
Serial.print(x6);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x7 = analogRead (A7);
Serial.print(x7);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x8 = analogRead (A8);
Serial.print(x8);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x9 = analogRead (A9);
Serial.print(x9);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x10 = analogRead (A10);
Serial.print(x10);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x11 = analogRead (A11);
Serial.print(x11);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x12 = analogRead (A12);
Serial.print(x12);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x13 = analogRead (A13);
Serial.print(x13);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x14 = analogRead (A14);
Serial.print(x14);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

int x15 = analogRead (A15);
Serial.print(x15);        // print the value in the serial monitor as a binary value
Serial.print (“,”);// to delimit bytes

Serial.print(‘\n’); // end of line

}

Processing Code: This is where the grid comes in…. We used the minim library in processing for the music.

import processing.serial.*;     // import the Processing serial library
import ddf.minim.*;

int grid[][] = new int[16][2];
int sensor[][] = new int[16][2];
boolean isPlaying[] = new boolean[16];
int isToggleSensor[] = {1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1};
boolean whiteOutPressed = false;

Minim minim;
AudioPlayer[] bobby;

Serial myPort;

int destination[] = {0, 0};
int currentPos[] = {0, 0};

int[] getNext(int[] c, int[] d)
{
int[] res = {0, 0};

res[0] = c[0];
res[1] = c[1];

if ((c[0] == d[0]) && (c[1] == d[1]))
return d;

float k = (float) (c[1] – d[1]) / (float) (c[0] – d[0]);
int b = d[1] – (int)(k * (float)d[0]);

if (abs(c[0] – d[0]) > abs(c[1] – d[1]))
{
res[0] -= ( (c[0] – d[0]) / abs(c[0] – d[0]) )/5 ;
res[1] = (int)((float)res[0] * k) + b;
}
else
{
res[1] -= ( (c[1] – d[1]) / abs(c[1] – d[1]) )/5 ;
res[0] = (int)(((float)(res[1] – b)) / k);
}

// println(“res = ” + res[0] + ” ” + res[1]);

return res;
}

void setup () {
size (600, 600);

// for (int i = 0; i < 4; i++)
// {
//   for (int j = 0; j < 4; j++)
//      {
//        grid[(i+1)*j][0] = (900 / 4) * (j + 1);
//        grid[(i+1)*j][1] = (900 / 4) * (i + 1);
//      }
//  }

grid[0][0] = 90;
grid[0][1] = 50;

grid[1][0] = 38;
grid[1][1] = 80;

grid[2][0] = 65;
grid[2][1] = 90;

grid[3][0] = 90;
grid[3][1] = 90;

grid[4][0] = 30;
grid[4][1] = 140;

grid[5][0] = 70;
grid[5][1] = 130;

grid[6][0] = 35;
grid[6][1] = 185;

grid[7][0] = 75;
grid[7][1] = 165;

grid[8][0] = 100;
grid[8][1] = 170;

grid[9][0] = 85;
grid[9][1] = 210;

grid[10][0] = 90;
grid[10][1] = 130;

grid[11][0] = 70;
grid[11][1] = 130;

grid[12][0] = 110;
grid[12][1] = 130;

grid[13][0] = 145;
grid[13][1] = 200;

grid[14][0] = 145;
grid[14][1] = 140;

grid[15][0] = 145;
grid[15][1] = 80;

smooth();
background(255);
colorMode(HSB);

minim = new Minim(this);
bobby = new AudioPlayer[16];

bobby[0] = minim.loadFile(“atmosphere.wav”, 1400);
bobby[1] = minim.loadFile(“clapwheel.wav”, 618);
bobby[2] = minim.loadFile(“anote.wav”, 1200);
bobby[3] = minim.loadFile(“bnote.wav”, 1200);
bobby[4] = minim.loadFile(“fastclick.wav”, 442);
bobby[5] = minim.loadFile(“d#.wav”, 975);
bobby[6] = minim.loadFile(“swishclick.wav”, 442);
bobby[7] = minim.loadFile(“g#.wav”, 975);
bobby[8] = minim.loadFile(“a4.wav”, 1200);
bobby[9] = minim.loadFile(“oceanbell.wav”, 180);
bobby[10] = minim.loadFile(“e.wav”, 1200);
bobby[11] = minim.loadFile(“f#.wav”, 975);
bobby[12] = minim.loadFile(“c#.wav”, 975);
bobby[13] = minim.loadFile(“mandolin.wav”, 709);
bobby[14] = minim.loadFile(“harmonic.wav”, 709);
bobby[15] = minim.loadFile(“underwaterclick.wav”, 618);

println(Serial.list());
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 9600);
myPort.bufferUntil(‘\n’);

}

void draw()
{
//  for (int i = 0; i < 16; i++)
//  {
//    fill(0);
//    ellipse(grid[i][0], grid[i][1], 20, 20);
//  }
//

if (whiteOutPressed)
{
background(255);
whiteOutPressed = false;
}
else
{
int[] next = getNext(currentPos, destination);

filter(DILATE);
filter(BLUR,6);

stroke(frameCount % 150, 200, 255);
strokeWeight(40);

line(currentPos[0], currentPos[1], next[0], next[1]);
currentPos[0] = next[0];
currentPos[1] = next[1];
}
}

void serialEvent (Serial myPort)
{
String input = myPort.readString().trim();
println(input);

if (input != null)
{
String [] parts = input.split(“,”);

if (parts.length != 16) //change to 8 when you put in the samples
{
return;
}

for (int i = 0; i < 16; i++)
{
if (parts[i] != null)
{
sensor[i][0] = sensor[i][1];
sensor[i][1] = int(map(int(parts[i]), 0, 300, 0, 1));
}
}

for (int i = 0; i < 16; i++)
{
if ( (sensor[i][0] == 0) && (sensor[i][1] == 1) )
{

if (isPlaying[i] && (isToggleSensor[i] == 1))
{
bobby[i].pause();
}
else
{
bobby[i].rewind();
if (isToggleSensor[i] == 1)
{
bobby[i].loop();
}
else
{
bobby[i].play();
}
}
isPlaying[i] = !isPlaying[i];

destination[0] = grid[i][0];
destination[1] = grid[i][1];

if (i == 9) // put the number of the whiteout sensor
{
currentPos[0] = destination[0] = 0;
currentPos[1] = destination[1] = 0;
whiteOutPressed = true;

}
}
}
}
}

 Project  Proposal Physical Computing:

Collaboration Project:  Ryan Viglizzo  and Veronika Dubrovskaya

Compose Music and Image with Light

An organ, 30 “ tall, built in a semicircle with 15  bamboo pipes will serve as an instrument to compose a musical peace along with image composition that will be projected on a large screen.  Each bamboo pipe will contain a photocell on its top.

Multiple flashlights will be provided to activate each one and many of the organ pipes.  When the light will shine on the photocell, it will produce a certain note or a collection of notes as well as particular imagery on the large screen.

Arduino analog input will determine position of each photocell (A0 – A14).

Further, the processing will map the readString from the arduino and map each output according to the design selected.

In addition, rwmidi library will be imported in processing to utilize Garage Band as music output, each analog output will be assigned a specific musical function.

We plan to prepare an original music composition peace with sequential sketching in processing  and perform for the class.  This would be a VJ/composer experience.  Further, any other members of the class will be encouraged to perform either by themselves or in collaboration with others on this  VJ/DJ organ.

This is the Arduino code for the prototype of four organ pipes:

Arduino:

void setup() {

Serial.begin(9600);

}

void loop() {

int x = analogRead (A0);       // read the pot value

Serial.print(x);        // print the value in the serial monitor as a binary value

Serial.print (“,”);// to delimit bytes

int x1 = analogRead (A1);

Serial.print(x1);        // print the value in the serial monitor as a binary value

Serial.print (“,”);// to delimit bytes

int x2 = analogRead (A2);

Serial.print(x2);        // print the value in the serial monitor as a binary value

Serial.print (“,”);// to delimit bytes

int x3 = analogRead (A3);

Serial.print(x3);        // print the value in the serial monitor as a binary value

Serial.print (“,”);// to delimit bytes

//int x4 = analogRead (A4);

//Serial.print(x4);        // print the value in the serial monitor as a binary value

// Serial.print (“,”);// to delimit bytes

Serial.print(‘\n’); // end of line

}

Processing:

import processing.serial.*;     // import the Processing serial library

import rwmidi.*;

int xpos[] = {0, 0}, xpos1[] = {0, 0}, xpos2[] = {0, 0}, xpos3[] = {0, 0};

//int notes [] = {0,2, 4,7, 12, 14,16, 19.21.24.26}

//int midiTonic = 60

MidiOutput output;

Serial myPort;

void setup () {

size (400, 400);

background (207, 86, 75);

println(Serial.list());

String portName = Serial.list()[0];

myPort = new Serial(this, portName, 9600);

myPort.bufferUntil(‘\n’);

output = RWMidi.getOutputDevices()[0].createOutput();

}

void draw()

{

background (207, 86, 75);

fill (255, 255, 255);

ellipse (xpos[1],  height/4, 40, 40);

ellipse (xpos1[1], 2*height/4, 40, 40);

ellipse (xpos2[1], 3*height/4, 40, 40);

ellipse (xpos3[1], 4*height/4, 40, 40);

}

void serialEvent (Serial mmyPort)

{

String input = myPort.readString().trim();

println(input);

int x = xpos[1];

int x1 = xpos1[1];

int x2 = xpos2[1];

int x3 = xpos3[1];

if (input != null)

{

String [] parts = input.split(“,”);

if (parts.length != 4)

{

return;

}

if (parts[0]!=null)

{

x = int (parts[0]);

}

if (parts[1]!=null)

{

x1 = int (parts[1]);

}

if (parts[2]!=null)

{

x2 = int (parts[2]);

}

if (parts[3]!=null)

{

x3 = int (parts[3]);

}

int aChannel = 1;

int aNote[] = {60,62,64,66};

int aVelocity = 127;

xpos[0] = xpos[1];

xpos1[0] = xpos1[1];

xpos2[0] = xpos2[1];

xpos3[0] = xpos3[1];

xpos[1] = int (map (x, 300, 1000, 0, width));

xpos1[1] = int (map (x1, 300, 1000, 0, width));

xpos2[1] = int (map (x2, 300, 1000, 0, width));

xpos3[1] = int (map (x3, 300, 1000, 0, width));

if ((xpos[1] > 30)  && (xpos[0] < 30))

{

output.sendNoteOn(aChannel, aNote[0], aVelocity); // note return a variable of type INT

}

if ((xpos1[1] > 30)  && (xpos1[0] < 30))

{

output.sendNoteOn(aChannel, aNote[1], aVelocity); // note return a variable of type INT

}

if ((xpos2[1] > 30)  && (xpos2[0] < 30))

{

output.sendNoteOn(aChannel, aNote[2], aVelocity); // note return a variable of type INT

}

if ((xpos3[1] > 30)  && (xpos3[0] < 30))

{

output.sendNoteOn(aChannel, aNote[3], aVelocity); // note return a variable of type INT

}

}

}

Phisical Computing Midterm  Media Controller

Graffiti Can

Graffiti Can Members
Adria Navarro-Lopez
Veronika Dubrovskaya
Melissa dela Merced
Joseph McCagherty

The goal was to create a graffiti can wich will remotely paint brush  over the images transmitted from webcam. Initially we wanted to only use an accelerometer to track the movement on the screen but that proved to be very challenging since there was a a lot of computational math involved in converting the acceleration of the sensor to actual X and Y points on the screen.
ITP Resident Greg Borenstein suggested that we should just track an LED light pointed at the camera in processing to create the images instead of using the accelerometer.

Blue Tooth was used to free the Graffiti Can.

Physical Computing Week 7 – Serial Communication Lab 2

This was the funnest lab!  In this lab the accelerometer is connected to two analog inputs.  Xtilt and Ytilt are used as serial inputs to create a visual string of white circles via the use of Arduino and Processing.  It is truly amazing to see the movement of the arm represented by a cute image on the screen of the computer.

  This is the photo of the computer screen that demonstrates structure of white circles which was created by simply moving the Arduino with the accelerometer up and down and from side to side.  This was a truly fun process.

This is the Arduino Code:

void setup() {
// initialize serial communications at 9600 bps:
Serial.begin(9600);
}

void loop(){
//int onOrOff = digitalRead(5); – if you have switch
// switchOut = 0;
// if digitalRead (ttt) switchOut +=1;
// if digitalRead (tt) switchOut +=2;
// if digitalRead (tt) switchOut +=4;
// if digitalRead (tt) switchOut +=8;
//Serial.print (onOrOff); – if you have switch
//Serial.print (“,”); – if you have switch
int x = analogRead (A0);
Serial.print(x);  // numbers or Serial.println
//Serial.prit (“X:”); – you can do this as well
/// Serial.write (x); –characters
Serial.print (“,”); // to seperate bytes
int y = analogRead (A1);
Serial.print(y);   // or Serial.println
Serial.print (“,”);

int z = analogRead (A2);
Serial.print(z);
Serial.print (“,”);// to delimit bytes
Serial.print(‘\n’); // end of line
//Serial.print (‘\t’) // tab

//Serial.write (10); – new line
//Serial.write (13);- carriage return

}
This is the Processing Code:

import processing.serial.*; Serial port; // The serial port
int xpos, ypos; // position of the ball

void setup()
{
size(800, 1000);  // Stage size
background (0, 100,200);
ellipseMode(CENTER);
xpos = width/2;
ypos = height/2;
// Print a list of the serial ports, for debugging purposes to find out what your ports are called:
println(Serial.list());
port = new Serial(this, Serial.list()[0], 9600);  //you can pull the name out of the list
}

void draw()
{
fill(255,255,255);

ellipse(xpos,ypos,30,30);
}

void serialEvent(Serial port)
{
String input = port.readStringUntil(‘\n’);  // you can use ‘\n’ or 10 make sure you print(“\n”) or write(10) at the end of your transmission in arduino
if (input != null)
{
print(“Raw Input: ” + input);
String[ ] parts = input.split(“,”);  //this will only work if you put commas (Ascii 44) between things in your transmission
if (parts.length > 2)
{ //make sure it is a full valid message, might have joined conversation in the middle of message
int xtilt = int(parts[0]);  //turn the “222″ into 222
int ytilt = int(parts[1]);
//int l = int(parts[2]);
xpos = int( map(xtilt,400,600,0,width) );  //you were putting this off in arduino but you have to do it some time
ypos = int( map(ytilt,400,600,0,height) );  //these will be used in draw function so you have to declared outside of this function.
//light = int( map(l,100,900,0,height) );  //these will be used in draw function so you have to declared outside of this function.
}
}
}

Here is the video: (via iphone on ITP floor)

accelerometer

Physical Computing Week 6 – Serial Communications  1

First attempt to use Processing in order to represent   Arduino’s analog output  via the use of  potentiometer.

Arduino Code:

void setup() {
Serial.begin(9600);
}
void loop() {
int analogValue = analogRead(A0) /4;      // read the pot value
Serial.write(analogValue);        // print the value in the serial monitor as a binary value

Cool Term Program was downloaded to interpret the data.

Processing Code:

import processing.serial.*;

Serial myPort;
int xpos;

void setup(){
println(Serial.list());
String partName=Serial.list()[0]; //string means text
myPort= new Serial (this, portName”put the name of modem instead for your prot”, 9600);
size (1800, 200);
stroke (255);
}

void draw(){
}

void serialEvent (Serial myPort) {
// get the byte:
int inByte = myPort.read();
// print it:
println(inByte);
}

/ at the edge of the screen, go back to the beginning:
if (xPos >= width) {
xPos = 0;
// clear the screen by resetting the background:
background(#081640);
}
else {
// increment the horizontal position for the next reading:
xPos++;
}

Here is the video:

serialinput

ToneBox

Tonebox is a small musical instrument that uses four photoresistors to produce four different musical notes. By projecting a flashlight on each photocell; the speaker produces a specific tone.

The goal is to make a large panel of photoresistors to have a musical intrument of many tones that responds to light stimulation.

Tone Box Picture

Tone Box Video Clips

1. toneboxclosed

2. toneboxopen

Arduino Code:

#define NOTE_C4  262

#define NOTE_D4  294

#define NOTE_E4  330

#define NOTE_F4  349

int sum = 0;

int threshold1 = 0;

int threshold2 = 0;

int threshold3 = 0;

int threshold4 = 0;

void setup() {

Serial.begin(9600);       // initialize serial communications

pinMode(3, OUTPUT);  // speaker

for (int i = 0; i < 10; i++) {

sum += analogRead(A0);

}

threshold1 = sum / 10;   //to take 10 waves of light and divide it by 10 for average

sum = 0;

for (int i = 0; i < 10; i++){

sum += analogRead(A1);

}

threshold2 = sum / 10;

sum = 0;

for (int i = 0; i < 10; i++){

sum += analogRead(A2);

}

threshold3 = sum / 10;

sum = 0;

for (int i = 0; i < 10; i++){

sum += analogRead(A3);

}

threshold4 = sum / 10;

}

void loop(){

int analogValue1 = analogRead(A0); // read the analog input

int analogValue2 = analogRead(A1); // read the analog input

int analogValue3 = analogRead(A2); // read the analog input

int analogValue4 = analogRead(A3); // read the analog input

// Serial.println(analogValue);      // print it

if (analogValue1 < threshold1 – 50) {

tone(3, NOTE_C4, 10);

}

if (analogValue2 < threshold2 – 50){

tone(3, NOTE_D4, 10);

}

if (analogValue3 < threshold3 – 50) {

tone(3, NOTE_E4, 10);

}

if (analogValue4 < threshold4 – 50){

tone(3, NOTE_F4, 10);

}

}

Physical Computing, Week 3

Lab: Electricity

PHYSICAL COMPUTING -   Arduino Analog Lab:

Week # 2

1.    Arduino Analog Lab:

arduino analog clip

2.   Sensors at Work:

3.  Fantasy Sensor Device:

Title:    ”Tunes with the Light”

Goal:    To create a device that will allow the movement of light to produce musical composition

This device will use a board of nine photo-resistors (3×3) that will be connected to  Arduino’s Analog input.  According to the strength of the lightwave and the location of the photo-resistor, the information will be processed by the digital output and send to MIDI device.  MIDI will have three components from the breadboard:  digital output, power and ground.  In turn, MIDI will connect to Mac Computer via USB port with the open Garage Band Program.  The GarageBand will be set to play and record the music (possibly on the Xylophone mode.)  The Arduino software will be used to program the specificities of each photo-resistor.  The Arduino microprocessor will be connected to the computer via another USB port.

In the future,  a larger scale project can consist of a corridor with beams of light coming form one wall onto the opposite wall, where the multitude of photo-resistors would be located. One could dance or walk or throw objects within the corridor and by his/her movement create music.