Archive for November, 2009

Moving Lights, cont’d

November 25th, 2009  |  Published in Introduction to Computational Media, Physical Computing

For my ICM final, I modified the ‘following a light’ sketch idea to displacing a projected abstract pattern: As a figure moves, the ground shifts. When an individual walks onto the floor, the surrounding sketch alters like this (use your mouse; from openprocessing):

The perspective shifting, illusory nature, and synthesis and articulation of planes of Op Art are influences in creating the projected pattern.

Without activation, the projection is still. Displacement will scrunch or widen shifting lines. While most the paintings are square or tiled, our projection will focus on the outline/line rather than the center (the grid as opposed to the squares).

For example, Victor Vasarely’s Vega:

Questions for ICM class feedback:

    How would I make the movement within the sketch realistic? For the simple sketch that allows the mouse to change the surrounding lines, translate, popMatrix, and pushMatrix are used.

On the Pcomp side, Joshua Clayton and I are using the physical computing/interactive aspect as our final in that class. We tested materials and methods to create a working sensor. Currently, we have a sensor made from wire mesh, and separated with 1/8 foam.

The next sensor will use wider foam strips so the area for closing the circuit will be smaller, as right now it’s very sensitive, and we’re likely to get false positives. Additionally, the plexiglass we’re considering using as the floor material is hard, and distributes its weight evenly across the sensor area. As a result, you have to apply quite a bit of pressure once the sensor is under a tile.

Sensor working when pressed:

Questions for class feedback:

    These tiles and underlying sensors have to be set up somehow with hidden wiring and Arduino. We’re considering a wooden box with cross bars for support. Thoughts or suggestions?
    Floor material. The floor is simultaneously the projection screen, the media controller, and the actual floor a person would stand on. Since we’re doing a scale model, we’re using our hands as our feet. Plexi looks great, but we’re finding it to require a good deal of pressure to give an ‘on’. Should we be considering softer materials?
Tags: , ,

Media Controller: Beat Feet

November 5th, 2009  |  Published in Physical Computing

Eric Mika, Arturo Vidich, and I finalized our mid-term media controller project. We made shoes that can produce music, or in this case, a combination of sounds written by the wearer’s dance steps.

Eric’s blog does a notably excellent and very thorough job documenting our progress:
Shoe Music

We ended up with a pair of shoes with four FSRs on the bottom of each. We mapped each FSR to a sound that would play when triggered, its volume dependent on the velocity with which it was hit.

The hat gives the user the ability to record, play back the recorded sound in a loop, and reset.

The interface was made initially to record which sensor was working, etc. It’s become another controller of sorts where the sensitivity threshold can be adjusted (e.g., how hard a sensor must be pressed in order to hear a sound) and where we can switch between all our fun libraries (twinkling, bubbles, piano, percussion, and noise samples).

A video demonstration:

The code is here: Beat Feet Code

Tags: , ,

Media Controller: Beat Feet Code

November 4th, 2009  |  Published in Introduction to Computational Media, Physical Computing


// Beat Feat
// Shoes that make sound.
// Yin Ho, Eric Mika, Arturo Vidich
// ITP Fall 2009

import processing.serial.*;
import controlP5.*;
import ddf.minim.*;
import arb.soundcipher.*;
import javax.swing.*;

Looper looper; // Handles cueing looped sequences in its own thread.
ControlP5 controlP5; // Sliders, buttons, etc.
Serial port; // Listen to the shoes.

Textlabel loopCount; // Keeps track of how many layers of loops we're in.

PImage schematic; // The background UI. Designed specifically for my laptop @ 1440 x 900. Will need modification for other resolutions.

String[] states; // Stores MIDI instrument configurations.
ListBox stateList; // Present instrument configurations in the UI.
Textfield stateTextfield; // Set filenames

// Channel Monitors.
int channelCount = 8; // 4 sensor channels per foot.
ChannelMonitor[] monitors = new ChannelMonitor[channelCount];

WeightDot[] dots = new WeightDot[channelCount]; // Show distribution of weight across sensors.

// Synth Library
SoundCipher sc; // SoundCipher has been crashy... looking for alternatives.
int instrument = 0; // (0 - 127)

// Playback options. Currently supports WAV samples or Java Sound Synth samples. Pure MIDI is coming.
int SAMPLE = 0;
int SYNTH = 1;

// Choose one of the above to use.
int playMethod = SYNTH;

void setup() {
size(1440, 900); // Only looks right at this resolution.
background(0);
frameRate(60);

sc = new SoundCipher(this);
schematic = loadImage("schematic.jpg");
// println(Serial.list()); // Show serial ports.
String portName = Serial.list()[0]; // The XBee Explorer FTDI chip is usually first in the list.
port = new Serial(this, portName, 115200); // 115k bps, With 2 XBees transmitting ~30byte packets at 10ms intervals we need at least 96kbps.

// Set up the weight dots.
// Left Foot.
dots[0] = new WeightDot(511, 57);
dots[1] = new WeightDot(470, 135);
dots[2] = new WeightDot(541, 122);
dots[3] = new WeightDot(499, 325);

// Right foot.
dots[4] = new WeightDot(720, 57);
dots[5] = new WeightDot(755, 135);
dots[6] = new WeightDot(684, 122);
dots[7] = new WeightDot(707, 325);

// Set up the channel monitors.
controlP5 = new ControlP5(this);

String[] channelNames = {
"Left Toe",
"Left Outside Ball",
"Left Inside Ball",
"Left Heel",
"Right Toe",
"Right Outside Ball",
"Right Inside Ball",
"Right Heel",
};

// Place the channel monitors accross the bottom of the window.
for (int i = 0; i < channelCount; i++) {
int w = round(width / channelCount); // Set width based on screen width.
int h = round(height * .5); // Half the screen height.
int xPos = w * i;
int yPos = height - h - 1;

// Fix rounding errors for the last channel monitor's width.
if (i == (channelCount - 1)) {
w = width - (w * (channelCount - 1)) - 1;
}

monitors[i] = new ChannelMonitor(channelNames[i], xPos, yPos, w, h);
}

// Set up the loop UI.
loopCount = new Textlabel(this,"x 1",60,15,400,200,color(0, 255, 0),ControlP5.synt24);
loopCount.setLetterSpacing(2);

// Fill the state list box.
stateList = controlP5.addListBox("stateList", width - 250,115,210,120);
stateList.setColorForeground(color(0, 200, 0));
stateList.setColorBackground(color(0, 80, 0));
stateList.setColorActive(color(0, 200, 0));
stateList.setItemHeight(15);
stateList.setBarHeight(15);
stateList.captionLabel().style().marginTop = 3;
stateList.valueLabel().style().marginTop = 3; // the +/- sign

// Load states from a folder and populate the list box.
refreshStateList();

// Set up save state filename textfield.
stateTextfield = controlP5.addTextfield("filename",width - 250,70,100,20);
stateTextfield.setColorActive(color(255));
stateTextfield.setColorForeground(color(0, 200, 0));
stateTextfield.setColorBackground(color(0, 80, 0));
stateTextfield.setCaptionLabel("");
stateTextfield.setText("File Name");

// Set up save state button.
controlP5.addButton("button",1,width - 140, 70, 100,20);
controlP5.controller("button").setColorActive(color(0, 200, 0));
controlP5.controller("button").setColorBackground(color(0, 80, 0));
controlP5.controller("button").setCaptionLabel("Save State");

// Jump trigger UI.
// Time sets how long the performer has to be in the air before we call it a jump.
Slider jumpTriggerTimeSlider = controlP5.addSlider("jumpTriggerTime", 0, 500, 160, width - 250, 30, 100, 10);
jumpTriggerTimeSlider.setColorForeground(color(0, 200, 0));
jumpTriggerTimeSlider.setColorActive(color(0, 200, 0));
jumpTriggerTimeSlider.setColorBackground(color(0, 80, 0));
jumpTriggerTimeSlider.setLabel("Jump Trigger Time");

// Weight sets how close to 0s the sensors need to read before we consider him to be in the air.
// (This compensates for any residual readings even if the shoes are completely unweighted.)
Slider jumpWeightThresholdSlider = controlP5.addSlider("jumpWeightThreshold", 0, 500, 300, width - 250, 50, 100, 10);
jumpWeightThresholdSlider.setColorForeground(color(0, 200, 0));
jumpWeightThresholdSlider.setColorActive(color(0, 200, 0));
jumpWeightThresholdSlider.setColorBackground(color(0, 80, 0));
jumpWeightThresholdSlider.setLabel("Jump Weight Threshold");

// Pick the sound files.
String[] soundFiles = {
"hey.wav",
"stuck.wav",
"crash.wav",
"bass.wav",

"fape.wav",
"tape.wav",
"drill.wav",
"earth1.wav",

"jump.wav",
};

// Create the looper object.
looper = new Looper(soundFiles, this);

smooth();
}

void draw() {
background(0);

// Draw the background image.
image(schematic, 0, 0);

noFill();
// Draw the Loop UI.
if (looper.recordingLoop) {
// Draw red circle if we're recording a loop.
stroke(255, 0, 0);
ellipse(30, 30, 40, 40);
}
else {
// Draw green "play" triangle if we're playing back a loop.
stroke(0, 255, 0);
triangle(10, 10, 10, 50, 40, 30);

if (looper.loops.length > 0) {
loopCount.setValue("X " + looper.loops.length);
loopCount.draw(this);
}
}

// Draw the channel monitors and weight dots.
for (int i = 0; i < channelCount; i++) {
monitors[i].draw();
dots[i].draw();
}
}

// Optionally press keys 0 - 8 to cue audio, useful for sound checks and debugging.
void keyPressed() {
// Go from ASCII to integer.
int numberKey = keyCode - 48;

// Do some bounds checking and play the sample.
if (numberKey >= 0 && numberKey < monitors.length) {
//println("Playing sample " + key);

int pitch = 0; // Always 0 for samples...

// If we're using the synth, get sample info from the associated channel.
if (playMethod == SYNTH) {
pitch = monitors[numberKey].getPitch();
numberKey = monitors[numberKey].getInstrument(); // Look out! this changes numberKey, so we can't use it for monitor indexing after this.
}

looper.playSample(numberKey, 0.0, 0.0, pitch, playMethod, false);
}
}

void keyReleased() {
if (key == ' ') {
// Space bar starts and stops looping for debugging and testing.
// Use keyReleased() instead of keyPressed() so it doesn't get fired in quick succession.

if (!looper.recordingLoop) {
looper.startRecording();
}
else {
looper.stopRecording();
}
}

if ((key == DELETE) || (key == BACKSPACE)) {
// Remove all the loops.
looper.deleteLoops();
}
}

// =================================================================================

// Receive serial data form the shoes.
// Since the XBees are in API mode and there are no Arduinos involved, we need
// to parse the XBee packets to extract the sensor data.
// The packet structure is documented here: http://ftp1.digi.com/support/documentation/90000866_C.pdf

// This should really be a library or class.
// If the XBee Library worked, that would be a better alternative.
// ( http://www.faludi.com/code/xbee-api-library-for-processing/ )

// "Frame" and "Packet" are used inconsistently and interchangeably in variable names.
int byteCounter = 0;
int frameIndex = 99999;
int frameLength;
int msb;
int lsb;
int[] frameArray;
int framesReceived = 0;

void serialEvent(Serial port) {
// println("Available: " + port.available());

// Grab the bytes as they arrive.
int latestByte = port.read();

// If the frame index is bigger than length and we received a frame-start byte (126), then start a new packet frame.
if ((latestByte == 126) && (frameIndex > (frameLength + 3))) {
frameIndex = 0; // Start the frame.

// Parse the bytes if we have a whole frame.
if (framesReceived > 0) parseFrame(frameArray);

framesReceived++;
}

// Byte 1 is the Most Significant Byte (MSB) of the checksum.
if (frameIndex == 1) msb = latestByte;

// Byte 2 is the Least Significant Byte (LSB) of the checksum.
if (frameIndex == 2) {
lsb = latestByte;

// Use the MSB and LSB to calculate the length of the frame.
frameLength = (msb << 8) + lsb;
frameArray = new int[frameLength];
}

// Add incoming bytes to the frame array (once we know how long it is).
// Making sure we've received a bunch of frames first seems to prevent start-up crashes.
if (((frameIndex > 3) && (framesReceived > 30)) && ((frameIndex - 4) < frameLength) && ((frameIndex - 4) >= 0)) {
// print(" " + (frameIndex - 4) + "/" + frameLength + " ");
frameArray[frameIndex - 4] = latestByte;
}

// print(frameIndex + ":" + nfs(latestByte, 3) + ' ');
frameIndex++;
}

// Now let's parse the XBee data packets...
int LEFT_SHOE = 16371; // 16 bit XBee ID for the left shoe.
int RIGHT_SHOE = 21258; // 16 bit XBee ID for the right shoe.
int HAT = 11880; // 16 bit XBee ID for the hat.

// Set up the hat buttons.
int lastHatButtonValue = 6;
int hatButtonValue = 0;
int lastHatPress = 0;

// Set up the shoes.
int shoeChannels = 4; // Channels per shoe.
int historyDepth = 2; // How many frames of analog data to keep for each channel. Useful for calculating sensor velocity.

// Set up jump detection. These values are tweaked at run-time via the UI.
int jumpStartTime = 0;
int jumpTriggerTime = 200;
boolean inAir = false;
int jumpWeightThreshold = 300;
int recordWeight = 0;

// Create an array to store the sensor data in an ugly 3 dimensional array.
// The structure is as follows:
// { Shoe data container
// { Left shoe
// { Channel 1
// { last reading, latest reading } (and so on, based on history depth)
// },
// { Channel 2
// { last reading, latest reading } (and so on, based on history depth)
// },
// { Channel 3
// { last reading, latest reading } (and so on, based on history depth)
// },
// { Channel 4 Container
// { last reading, latest reading } (and so on, based on history depth)
// }
// },
// { Right shoe
// { Channel 1
// { last reading, latest reading } (and so on, based on history depth)
// },
// { Channel 2
// { last reading, latest reading } (and so on, based on history depth)
// },
// { Channel 3
// { last reading, latest reading } (and so on, based on history depth)
// },
// { Channel 4
// { last reading, latest reading } (and so on, based on history depth)
// }
// }
// }

int[][][] shoeData = new int[2][shoeChannels][historyDepth];
int shoeIndex = 0;

// Extract sensor data from the XBee packet.
void parseFrame(int[] frame) {

// Get the 16 BIT address, so we'll know where to send the sensor data.
int address = (frame[8] << 8) + frame[9];
// print("address 16:" + address + " ");
// printArray(frame);

if (address == HAT) {
// Check the button values. This is a bit different from the shoes since it's
// digital data, not analog. (With the exception of the hat lift sensor, which is broken.)
hatButtonValue = frame[16];

// println ("hatButtonValue " + hatButtonValue + " lastHatButtonValue " + lastHatButtonValue);
// println ("time since: " + (millis() - lastHatPress));

// Detect the loop start button (debounced via lastHatPress)
if ((hatButtonValue == 4) && (lastHatButtonValue == 6) && ((millis() - lastHatPress) > 200)) {
lastHatPress = millis();

// Toggle looping.
if (!looper.recordingLoop) {
looper.startRecording();
}
else {
looper.stopRecording();
}
}

// Detect the reset button. This one isn't debounced since it can fire as much as it wants to without incident.
if ((hatButtonValue == 2) && (lastHatButtonValue == 6)) {
// Remove all the loops.
looper.deleteLoops();
}

lastHatButtonValue = hatButtonValue;

// Check for hat liftoff.
// Lifting the hat off of one's head was supposed to trigger something, but the hardware
// never worked correctly so this aspect was scrapped. Here's some code in case we ever fix it.
// int hatBrightness = (frame[17] << 8) + frame[18];
// println("hatBrightness: " + hatBrightness);
// TK check threshold
// TK send to monitor
}

// Handle shoe data.
if (address == LEFT_SHOE || address == RIGHT_SHOE) {

// Read the four analog values, which are two bytes each... so shift them bits!
int[] analogValues = {
(frame[15] << 8) + frame[16],
(frame[17] << 8) + frame[18],
(frame[19] << 8) + frame[20],
(frame[21] << 8) + frame[22],
};

// Set the shoe index for storage in the big hairy 3D array.
// Left is 0, right is 1.
shoeIndex = 0; // Default to left shoe.
if (address == RIGHT_SHOE) shoeIndex = 1; // Switch to right if we need to.

// Load up the data for each channel, keep specified amount of sensor history.
for (int i = 0; i < shoeChannels; i++) {
// Add the latest value to the end of the array.
shoeData[shoeIndex][i] = append(shoeData[shoeIndex][i], analogValues[i]);

if (shoeData[shoeIndex][i].length > historyDepth) {
// Pop a value off the start of the history array to keep it the right length.
shoeData[shoeIndex][i] = subset(shoeData[shoeIndex][i], 1);
}

// Find velocity by looking at several sensor values over time.
int velocity = findVelocity(shoeData[shoeIndex][i]);
int analogValue = analogValues[i];
// println("velocity: " + findVelocity(shoeData[shoeIndex][i]));

// Update the channel monitors and weight dots.
int monitorIndex = (shoeIndex * shoeChannels) + i;
int lastValueIndex = monitors[monitorIndex].valueIndex - 1;

if (lastValueIndex < 0) {
// Does this work?
lastValueIndex = monitors[monitorIndex].values.length - 1;
// println(monitors[monitorIndex].accelerations[lastValueIndex]);
}

monitors[monitorIndex].setAcceleration(velocity);
monitors[monitorIndex].setValue(analogValue);
dots[monitorIndex].value = analogValue;

// Find total weight on the left foot...
int leftTotal = 0;
for (int j = 0; j < shoeData[0].length; j++) {
leftTotal += shoeData[0][j][shoeData[0][j].length - 1];
}

// Find total weight on the right foot...
int rightTotal = 0;
for (int j = 0; j < shoeData[1].length; j++) {
rightTotal += shoeData[1][j][shoeData[1][j].length - 1];
}

// Find the total weight on both feet.
// We need this to look for jumps.
int totalWeight = leftTotal + rightTotal;
// println("total weight: " + totalWeight);

// Keep track of the record weight.
if (totalWeight > recordWeight) recordWeight = totalWeight;

// Look at left-right balance.
float balance = 0.0;

// Only take the balance if we're ostensibly on the ground.
// This gets used later to set the pan of the sound.
if (totalWeight > jumpWeightThreshold) {
balance = map((float)leftTotal / ((float)rightTotal + (float)leftTotal), 0, 1, -1, 1);
}

//println("footBalance: " + balance);
//println("jumpWeightThreshold " + jumpWeightThreshold);

// Check for Jumps, change back to 300.

// Make sure we've had our feet on the ground before thinking about jumps.
if (((totalWeight < jumpWeightThreshold) && !inAir) && (recordWeight > jumpWeightThreshold)) {
// Start the timer, if we need to.
if (jumpStartTime == 0) {
jumpStartTime = millis();
}

// Call it a jump if we've been in the air long enough.
if (((millis() - jumpStartTime) < jumpTriggerTime)) {
jumpStartTime = 0; // Reset the start time.
inAir = true;
int jumpSoundIndex = 8; // Play the 9th sample (index 8).
looper.playSample(jumpSoundIndex, 0.0, 0.0, 0, SAMPLE, false);
}
}

// Reset the inAir boolean if we're on the ground.
if (totalWeight >= jumpWeightThreshold) inAir = false;

// Cue the samples.

// Control sample volume with velocity. Harder stomps = louder sounds.
if ((velocity > monitors[monitorIndex].threshold) && (monitors[monitorIndex].accelerations[lastValueIndex] < monitors[monitorIndex].threshold)) {
// Set the gain depending on how far we passed the threshold.
// Catch the hardest press on each channel to calibrate this dynamically.
if (velocity > monitors[monitorIndex].peak) monitors[monitorIndex].peak = velocity;

// Distance over the threshold determines loudness.
int distance = abs(monitors[monitorIndex].threshold - velocity);
if (distance > monitors[monitorIndex].biggestDistance) monitors[monitorIndex].biggestDistance = distance;

// Set the gain accordingly.
float gain = map(distance, 0, monitors[monitorIndex].biggestDistance, -10, 10);

// TK move the play type to the channel monitor
if (playMethod == SYNTH) {
looper.playSample(monitors[monitorIndex].getInstrument(), balance, gain, monitors[monitorIndex].getPitch(), playMethod, false);
}

if (playMethod == SAMPLE) {
looper.playSample(monitorIndex, balance, gain, 0, playMethod, false); // 0 pitch for now
}

// Turn one of the dots red momentarily to show that it triggered.
dots[monitorIndex].showHit();
}
}

}
}

// Helper function to determine velocity based on the history array.
int findVelocity(int[] values) {
int firstValue = values[0];
int lastValue = values[values.length - 1];
int elapsed = values.length;

return round((lastValue - firstValue) / elapsed);
}

// Helper function to help inspect XBee packets.
void printArray(int[] packetArray) {
for (int i = 0; i < packetArray.length; i++) {
print(nfs(packetArray[i], 3) + ' ');
}
println();
}

// =================================================================================

// Synth state management.
// This should also live in its own class. Never got around to it.

// Load a state file.
// A folder called "states" int he Processing sketch's "data" folder
// should containe plain text files with the following syntax. Each row
// accounts for a channel between 1 and 8.

// Pitch Instrument
// 64 4
// 52 65
// 53 1
// 46 3
// 63 4
// 33 55
// 63 0
// 23 4

void loadState(String path) {
// println("Loading State " + path);
String lines[] = loadStrings(path);

// Parse the state file.
// Ignore the first line, send the rest to the channel monitors to save state
for (int i = 1; i < lines.length; i++) {
String values[] = split(lines[i], "\t");
// In format
// pitch instrument
monitors[i - 1].setPitch(Integer.parseInt(values[0]));
monitors[i - 1].setInstrument(Integer.parseInt(values[1]));
}
}

// Save state button.
void button(float theValue) {
if (theValue == 1.0) {
// save the state
saveState();
refreshStateList();
}
}

// Write a new text file with the current instrument / pitch settings for each channel.
void saveState() {
String[] state = new String[monitors.length + 1];

state[0] = "Pitch\tInstrument";
for (int i = 1; i <= monitors.length; i++) {
state[i] = monitors[i - 1].getPitch() + "\t" + monitors[i - 1].getInstrument();
}

String fileName = stateTextfield.getText() + ".txt";
saveStrings("states/" + fileName, state);
// println("Saved State in file " + fileName);
}

// Re-reads the states folder to detect any new files.
void refreshStateList() {
// If we already have the list, clear it first.
if (states != null) {
for (int i = 0; i < states.length; i++) {
stateList.removeItem(states[i]);
}
}

File dataDir = new File(sketchPath, "states");
File[] files= dataDir.listFiles();
states = new String[files.length];
for (int i=0; i states[i] = files[i].getName();

// Hide hidden files.
if (files[i].getName().charAt(0) != '.') {
stateList.addItem(files[i].getName(), i);
}
}
}

// Load saved states through the UI.
void controlEvent(ControlEvent theEvent) {
if (theEvent.isGroup()) {
// An event from a group e.g. scrollList
// Load that file that was licked on.
String filename = sketchPath + "/states/" + states[round(theEvent.group().value())];
println("Loading state " + filename);
loadState(filename);
}
}

class Looper implements Runnable {
// Custom class uses Minim to record and play back loops of samples.
// Creates its own thread since the draw() loops is too slow to replay
// rapid sequences of samples. The main loop in this class runs much,
// much faster than the draw() loop.

// This class also manages basic playback of non-looped audio samples.

// This class build a confusing, three-deep structure of nested loops.
// Note that the last frame of each sample loop holds metadata and state
// information.

// Here's the structure:
// { start loops
// { // start loop
// {time, sample, balance, gain, pitch, type}
// {time, sample, balance, gain, pitch, type}
// ... additional samples ...
// {time, sample, balance, gain, pitch, type}
// {time, sample, balance, gain, pitch, type}
// {loop duration, loop start time, loop count, current index} // meta data...
// } // end loop
// } // end loops

Minim minim;
String[] soundFiles;
AudioSample[] samples;
boolean recordingLoop = false;
int[][][] loops = new int[0][0][0];
int[][] workingLoop = new int[0][0];
int loopStartTime;
Thread looperThread;
PApplet parent;

// Constructor. Takes an array of audio filenames and the parent applet.
Looper(String[] _soundFiles, PApplet _parent) {
// println("Instantiating a new Looper.");
soundFiles = _soundFiles;
parent = _parent;
minim = new Minim(parent);

// Make an array to hold the samples that's the same length as the file list.
samples = new AudioSample[soundFiles.length];

// println(soundFiles.length);

// Preload each sound file into memory.
for(int i = 0; i < soundFiles.length; i++) {
samples[i] = minim.loadSample(soundFiles[i]);
println("Loaded sample " + soundFiles[i] + ".");
}

// Create the thread supplying it with the runnable object.
looperThread = new Thread(this);

// Start the thread.
looperThread.start();
}

// Play a sample.
void playSample(int sampleIndex, float balance, float gain, int pitch, int type, boolean autoPlayback) {
// auto playback is true if we're playing the sample from a loop, which means we shouldn't record it again

if(type == SAMPLE) {
samples[sampleIndex].setPan(balance);
samples[sampleIndex].setGain(gain);
samples[sampleIndex].trigger();
}

if(type == SYNTH) {
sc.instrument(sampleIndex); // Use sample index to store instrument.
sc.pan(64); // 0 - 127
sc.playNote(pitch, 64, 0.2); // Pitch, Volume, Duration.
}

// Write it down if we're recording.
if(recordingLoop && !autoPlayback) {
// Create a new array entry with the time, and then the key pressed.
int[] currentSample = {millis() - loopStartTime, sampleIndex, floatToInt(balance), floatToInt(gain), pitch, type};

// Then add it to the end of the latest active loop.
workingLoop = (int[][])append(workingLoop, currentSample);
}
}

int getSampleCount() {
return samples.length;
}

// Delete stops and destroys all loops.
void deleteLoops() {
// Make sure any currently recording loops are stopped.
if (recordingLoop) {
// println("Stopping current recording loop.");
stopRecording();
}

// Clear the loops array.
// println("Deleting all loops.");
loops = new int[0][0][0];
}

void startRecording() {
// println("Start recording loop.");
recordingLoop = true;

// Clear the working loop.
workingLoop = new int[0][0];

// Note when we started.
loopStartTime = millis();
}

// Stops recording and starts playing it back.
void stopRecording() {
// println("Stop recording loop, play it back!");
// Could make playback a separate function, might be cleaner that way.

// Create and fill out the metadata frame of the loop.
// {loop duration, loop start time, loop count, current index}
int[] end = { millis() - loopStartTime, loopStartTime, 0, 0 };

// Tack this metadata frame onto the end of the loop.
workingLoop = (int[][])append(workingLoop, end);

// Add the working loop to the list of playing loops.
loops = (int[][][])append(loops, workingLoop);

// println("That's loop layer number " + loops.length + ".");

recordingLoop = false;
}

// Thread to playback any loops.
// This class doesn't record any waveforms, it just notes when each sample
// was played. When looping, it skims through the lists of samples and plays
// them according to their relative position in time.
void run() {

// Always running.
while(true) {

// Go through each sound loop and position the playhead relative to the clip, and play what we need
// to if time is equal to or greater than sample time (assuming we have actually made a loop).
for(int i = 0; i < loops.length; i++) {

// See if there's anything to play at the front of the list.
int loopDuration = loops[i][loops[i].length - 1][0];
int loopStartTime = loops[i][loops[i].length - 1][1];
int lastLoopCount = loops[i][loops[i].length - 1][2];
int nextIndex = loops[i][loops[i].length - 1][3]; // Which sample to play.

int currentLoopCount = (millis() - loopStartTime) / loopDuration; // Number of times this loop has played.
loops[i][loops[i].length - 1][2] = currentLoopCount;

if(currentLoopCount > lastLoopCount) {
// Starting a new loop, move the next index back to start.
nextIndex = 0;
loops[i][loops[i].length - 1][3] = nextIndex;
}

int nextTime = loops[i][nextIndex][0];
int nextNote = loops[i][nextIndex][1];
int playHead = (millis() - loopStartTime) % loopDuration;

// Play the note as we land on it or pass it.
if (playHead >= nextTime) {
float balance = intToFloat(loops[i][nextIndex][2]);
float gain = intToFloat(loops[i][nextIndex][3]);
int pitch = loops[i][nextIndex][4];
int type = loops[i][nextIndex][5];

// make the past a bit quieter
gain -= 3;

playSample(nextNote, balance, gain, pitch, type, true);

loops[i][loops[i].length - 1][3]++; // Move to the next index.
}
}

// Pause for a millisecond so we don't hog the CPU.
try {
looperThread.sleep(1);
}
catch(InterruptedException e) {
e.printStackTrace();
}
}
}

// Ugly hack so we can stick with ints when we need to store balance and gain info.
// This is criminal, really. Stores floats as ints by multiplying out the decimal point.

// This turns floats into ints (retains up to 4 decimal places).
int floatToInt(float input) {
return round(input * 10000.0);
}

// This turns ints into floats (restores up to 4 decimal places).
float intToFloat(int input) {
return (float)((float)input / 10000.0);
}

// Clean up after Minim. Is this actually getting fired?
void stop() {
// Always close Minim audio classes when you are done with them.
for(int i = 0; i < samples.length; i++) {
samples[0].close();
}

minim.stop();
}
}

class ChannelMonitor {
// Provides graphical representation of the 8 analog sensor channels.

int xPos, yPos, w, h, currentValue, threshold;
// int maxValue = round(1024 * 0.7); // Corrects for low voltage, we are using 1.0 v where we should use 1.2...
int maxValue = 0; // Just calibrate dynamically instead since velocity is more important.
int valueIndex = 0;

boolean newValue;
String channelName;
Textlabel channelLabel;
int sweepDistance;

int biggestDistance = 0; // Farthest we've ever gone past the threshold.
int peak = 0;

Toggle showValues;
Toggle showAccelerations;
Slider instrumentSlider;
Slider pitchSlider;

// Store sensor and velocity values.
int[] values;
int[] accelerations;

// Constructor
ChannelMonitor(String _channelName, int _xPos, int _yPos, int _w, int _h) {
channelName = _channelName;
xPos = _xPos;
yPos = _yPos;
w = _w;
h = _h;

// Start with the threshold just below the top.
// Would be nice if this was stored accross sessions.
threshold = h - 15;

channelLabel = controlP5.addTextlabel(channelName, channelName, xPos + 3, yPos - 10);
channelLabel.setWidth(w - 26);

// Toggle switches control which data is shown in the monitor (straight values are green, velocities are red.)
showValues = controlP5.addToggle("", false, xPos + w - 13, yPos - 12, 10, 10);
showValues.setColorActive(color(0, 255, 0));
showValues.setColorForeground(color(0, 80, 0));
showAccelerations = controlP5.addToggle("",true,xPos + w - 26,yPos - 12,10,10);
showAccelerations.setColorActive(color(255, 0, 0));
showAccelerations.setColorForeground(color(80, 0, 0));

// Instrument slider, selects which synth instrument to play. (Doesn't apply to samples.)
instrumentSlider = controlP5.addSlider("channelInstrument", 0, 127, 0, xPos, yPos - 40, 127, 10);
instrumentSlider.setColorForeground(color(0, 200, 0));
instrumentSlider.setColorActive(color(0, 200, 0));
instrumentSlider.setColorBackground(color(0, 80, 0));
instrumentSlider.setLabel("Instrument");

// Pitch slider, sets the sytnth instrument's pitch. (Doesn't apply to samples.)
pitchSlider = controlP5.addSlider("channelPitch", 0, 127, 64, xPos, yPos - 60, 127, 10);
pitchSlider.setColorForeground(color(0, 200, 0));
pitchSlider.setColorActive(color(0, 200, 0));
pitchSlider.setColorBackground(color(0, 80, 0));
pitchSlider.setLabel("Pitch");

// TK Velocity Sliders.

values = new int[w];
accelerations = new int[w];

// println("Instantiating Channel Monitor");

// Erase any values already logged in the monitor.
clear();
}

void draw() {
clear();

// Set the threshold level with the mouse.
if(mousePressed && mouseInside()) {
// println("Clicking on channel " + channelName);
threshold = (yPos + h) - mouseY;
}

// Draw the values.
for(int i = 0; i < values.length; i++) {

if(i < valueIndex) {
sweepDistance = i + (values.length - valueIndex);
}
else {
sweepDistance = i - valueIndex;
}

// Fade out the older values.
int opacity = round(map(sweepDistance, 0, values.length * 1.5, 0, 255));
// int opacity = 255; // Uncomment to disable fade.

// Just erase on home (and don't show first value for now).
// if((i <= valueIndex) && (i != 0)) {

// Draw the sensor values.
if(showValues.value() == 1.0) {
stroke(0, 255, 0, opacity);
line(i + xPos, yPos + h - 1, i + xPos, (yPos + h - 1) - constrain(map(values[i], 0, maxValue, 0, h), 0, h - 2));
}

// Draw the velocity / acceleration values.
if(showAccelerations.value() == 1.0) {
stroke(255, 0, 0, opacity);
line(i + xPos, yPos + h - 1, i + xPos, (yPos + h - 1) - constrain(accelerations[i], 0, h - 2));
}

//}
}

// Draw the threshold bar.
stroke(0, 255, 0, 255);
line(xPos + 1, h + yPos - threshold, xPos + w, h + yPos - threshold);

// Draw the peak level bar.
stroke(255, 0, 0, 100);
line(xPos + 1, h + yPos - peak, xPos + w, h + yPos - peak);

}

// Clears out the sensor area.
void clear() {
fill(0, 255);
stroke(255, 255);
rect(xPos, yPos, w, h);
rect(xPos, yPos, w, - 15);
}

// Returns true if the mouse click was inside this channel.
boolean mouseInside() {
if((mouseX > xPos && mouseX < xPos + w) && (mouseY > yPos && mouseY < yPos + h)) {
return true;
}
else {
return false;
}
}

// Helper to set threshold. Could implement some error checking.
void setThreshold(int value) {
threshold = value;
}

// Helper to get instrument.
int getInstrument() {
return round(instrumentSlider.value());
}

// Helper to set instrument. Could implement some error checking.
void setInstrument(int value) {
instrumentSlider.setValue(value);
}

// Helper to get pitch.
int getPitch() {
return round(pitchSlider.value());
}

// Helper to set instrument. Could implement some error checking.
void setPitch(int value) {
pitchSlider.setValue(value);
}

// Store the sensor values.
void setValue(int value) {
if (valueIndex > values.length - 1) valueIndex = 0;

if(value > maxValue) maxValue = value;
values[valueIndex] = value;
valueIndex++;
}

// Store the sensor velocities.
void setAcceleration(int acceleration) {
if (valueIndex > values.length - 1) valueIndex = 0;
accelerations[valueIndex] = acceleration;
}

// Didn't end up using this desaturation function.
color desaturate(color c) {
return color(red(c) * 0.8, green(c) * 0.8, blue(c) * 0.8);
}
}

class WeightDot {
// Circles to show how much weight is on a particular channel.

int xPos, yPos, diameter;
int maxValue = 500;
int value = 0;
int minDiameter = 20; // Diameter at rest.
int maxDiameter = 70; // Largest possible diameter.
int redness = 255;

// Constructor
WeightDot(int _xPos, int _yPos) {
xPos = _xPos;
yPos = _yPos;
}

void draw() {
// Fade from red to white when this
if(redness < 255) redness += 5;

// Draw the circle.
noStroke();
fill(255, redness, redness);

diameter = round(map(value, 0, maxValue, minDiameter, maxDiameter));
ellipse(xPos, yPos, diameter, diameter);
}

// Trigger a hit.
void showHit() {
redness = 0;
}
}

Tags: , ,

Recurring Concepts Midterm: River

November 4th, 2009  |  Published in Recurring Concepts in Art

I worked with Melissa Clarke on our midterm project, River. The assignment was to take a technological piece one of us had made and re-create it without technology.

The piece we made was based on Melissa’s video installation Acoustic Imaging. The original piece was inspired by the visuals generated using acoustic mapping technology by the Hudson River Benthic Mapping Project. Sound is used to reflect surface/subsurface depth. The original piece plays with the data visualization provided by sonar waves while playing a modified recording of the source material used to locate the riverbed.

Glass beads and magnetic filings create a formal topography that can be shifted and altered by magnets affixed to the top of the plexiglass. The transparency of the material affords a vantage point to ‘seeing’ the submerged earth, changing from each perspective. The suspension of the tube articulates that the river bottom as a base and lowest point is not truly that: rather, it is another layer of earth. We’re afforded a fanciful view of that insight by being able to observe the grains from underneath.

The potential for flux lies inherent in the piece. Observers alter the interior beads and filings as they move the magnets. As magnetic force forms peaks and troughs, the material distribution reflects the sonar measurement of depth, and the riverbed itself. Sound captures the shifting landscape by offering a reading, bounded temporally in that specific instant of the wave reflected on a particular spatial point. The stability of that point in time and space is an illusion; constant, often imperceptible, flux is always occurring. The reading, then, is an approximation. The sound of beads and filings moving highlight the impossibility of a truly accurate observation: as you hear the sound, the measured ground has already been displaced. It is both a moving target, and the source of the sound. This quest for accuracy in measurement highlights a remarkably human search for understanding and truth through empirical observation within a constant state of change.

Tags: , ,