Category Archives: Dataflow Programming

Improvisations

Posted on by
Reply

For the past few weeks I’ve been exploring the relationship between motion and sound through the medium of dance, with the intention of creating a more meaningful link between the visual and sonic aspects of the performance. Historically choreography has been grafted onto existing pieces of music that were written in a completely separate context, or music was scored with the choreography in mind but was still performed by musicians, not dancers. In many cases a conductor has served the role as a link between the two. I can’t necessarily argue that the repertoire has suffered because of this disconnect, but I though it would be worthwhile to try to eliminate the middleman and tether the score directly to the dancers. The following improvisations contain sound elements that react to the motion of the dancer’s body in real time, generating the score on the spot. I was fortunate to work with the super talented dancer Jennifer Landa on this project. My apologies for the focus issues in the video.

The biggest challenge I faced was the lack of rehearsal time with Jen. There’s only so much you can predict without actually getting in a room with the performer and testing. I had a couple weeks to work before our first rehearsal so I wanted to get the entire architecture done beforehand, which would leave me with another week to tweak it before our final shoot. First I set about putting my sounds together. I decided to create three layers of sound. One layer would be a drone that remains constant and is not controlled by the dancer in any way. For this I chose a low, rumbly, drone which provides some substance but leaves plenty of room for other sounds to skate above it. Second is a layer of sound that responds to motion in general. Here I used some filters without any audio going through them. By manipulating the parameters of the filters I was able to create some glitches that I find very appealing. In the recording you can hear this layer as the ghosty pitches and watery glitches. The position of Jen’s feet are controlling this layer. The third layer of sound are samples that are triggered by specific gestures. For this I’m using some bleeps and a synth stab. The final audio component is a system that mimics the sound of electricity. I wanted to have the electricity gain intensity as two dancers moved toward each other. Unfortunately I couldn’t get a second dancer so instead I set a hotspot, in the video toward the right front of the stage. This starts a couple of minutes into the video.

In the future I would like to incorporate some projections into this project, so with that in mind I am using Processing to take in the skeleton data from the Kinect. From there I send OSC messages to Pd, where the audio is generated based on the input. I built a series of sub patches to parse the OSC messages, which make it easy for me to nimbly switch the data being sent in Processing and interpret and route the data in Pd. This flexibility was especially important because I knew there was going to be a lot of trial and error and not a lot of time when I got in the studio with Jen. The initial iteration mapped hand position to the filter layer and used a couple of hand gestures to trigger the samples.

A big challenge in the first rehearsal was calibrating to the space and to Jen’s body. Up until that time I had been testing in my cramped apartment and in cramped rooms at school. Also, it became apparent pretty quickly that Jen’s movements were completely different from the mine. I had never thought about rolling around on the ground for instance. The Kinect had some trouble keeping track of her. I think it was great for Jen to get some time working with the setup because it gave her ideas as far as what types of movement would be easy for the Kinect to interpret. Generally, the slower she moved the more reliable it was. And she started learning how to prevent inadvertent triggers of the samples. At the end of the rehearsal we decided to switch the filtered layer of sound to her legs because she tends to focus a lot of her moves there. We were also able to test the electricity layer in a basic way, but there was still some work to do mapping that data in an effective way.

First rehearsal
Jennifer Landa in the studio

In the week between the rehearsal and the final shoot I worked on expanding the system for multiple dancers, increasing reliability, fine tuning my data mapping, and developing the sounds. I was pushing the system to the limit so I had to pick and choose what data to send at any given time. The day of the shoot we spent some time testing and fine tuning. I could see how Jen’s level of control improved as time went on. For the final take Jen went non-stop for 45 minutes! The Kinect lost her a couple times and my filters exploded a few times too (I tried using a limiter, but it wasn’t fool proof). I plan on editing the footage some more and putting up more excerpts.

Zips and Zaps

Posted on by
Reply

Continuing work on my dance scoring project I found a patch that synthesizes the sound of electricity and began working with it. The goal was to map the intensity of the zips and zaps to the distance between two dancers. I was able to hook my skeleton data from the Kinect/Processing side into the patch to get it working in a basic way. I also added some reverb to fill out the sound. I’ve also built out a more robust system to handle the OSC messages. Download my work in progress patch.

Body Music

Posted on by
Reply

Building off of Greg Borenstein’s skeleton recorder example for Processing and Kinect, I created a musical interface that uses skeleton joint position/movement to generate and/or affect audio. For this example only the hands are used. I’m using a Pd patch derived from Damian Stewart’s Dinosaurs for the audio and OSC to communicate between Processing and Pd. The movement of the hands controls a low pass filter that generates some interesting glitches and bleeps on it’s own, and can also be used to filter audio generated by a sequencer. As with Greg’s example, the position of the joints are recorded and can be used to play back the previous performance. I set up the playback mode so the recorded performance is passed through a filter controlled by the current joint positions, essentially remixing the previous performance. For now only the x position of each hand is being used but I would like to build it out further, using more joints and mapping sound events to more complex gestures rather than simply the position of an individual joint. I could see this as the audio component of a dance or live multimedia performance… which will probably be my final project.

I couldn’t get the Kinect depth image working with this one so the video isn’t ideal to see how this works. The blue sphere is the right hand and the red sphere is the left hand. The video starts in the record mode without the sequencer, then adds the sequencer, and finally the playback mode.

Download the Pd patch

Here’s the Processing code:

import processing.opengl.*;
import SimpleOpenNI.*;
import oscP5.*;
import netP5.*;

SimpleOpenNI kinect;

SkeletonRecorder recorder;

boolean recording = false;
boolean playing = false;

float offByDistance = 0.0;
PFont font;

OscP5 oscP5;
NetAddress myRemoteLocation;

void setup()
{
size(1028, 768, OPENGL);
kinect = new SimpleOpenNI(this);
kinect.enableDepth();
kinect.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);
kinect.setMirror(true);

// initialize our recorder and
// tell it to track left hand
// it takes an array because it can track multiple joints
int[] jointsToTrack = {SimpleOpenNI.SKEL_LEFT_HAND, SimpleOpenNI.SKEL_RIGHT_HAND};
recorder = new SkeletonRecorder(kinect, jointsToTrack);

font = createFont(“Verdana”, 40);
textFont(font);

oscP5 = new OscP5(this, 12004);
myRemoteLocation = new NetAddress(“127.0.0.1″, 12006);

}// end setup

void draw()
{
background(0);
kinect.update();
// display text information
pushMatrix();
// scale(4);

fill(255);
translate(0, 50, 0);
text(“totalFrames: ” + recorder.totalFrames, 5, 0);
text(“recording: ” + recording, 5, 50);
text(“currentFrame: ” + recorder.currentFrame, 5, 100 );
float c = map(offByDistance, 0, 1000, 0, 255);
fill(c, 255-c, 0);
//text(“off by: ” + offByDistance, 5, 150);
popMatrix();

translate(width/2, height/2, 0);
rotateX(radians(180));

IntVector userList = new IntVector();
kinect.getUsers(userList);
if (userList.size() > 0)
{
int userId = userList.get(0);
recorder.setUser(userId);
if ( kinect.isTrackingSkeleton(userId))
{
PVector leftHandPosition = new PVector();
kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_LEFT_HAND, leftHandPosition);

PVector convertedLeftHandPosition = new PVector();
kinect.convertRealWorldToProjective(leftHandPosition, convertedLeftHandPosition);

PVector rightHandPosition = new PVector();
kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_HAND, rightHandPosition);

PVector convertedRightHandPosition = new PVector();
kinect.convertRealWorldToProjective(rightHandPosition, convertedRightHandPosition);

pushMatrix();
stroke(255, 0, 0);
strokeWeight(50);
point(leftHandPosition.x, leftHandPosition.y, leftHandPosition.z);
stroke(0, 0, 255);
point(rightHandPosition.x, rightHandPosition.y, rightHandPosition.z);
popMatrix();

String messageText = “/recording/lefthand/x/” + convertedLeftHandPosition.x;
//println(messageText);
OscMessage triggerMessage = new OscMessage(messageText);
oscP5.send(triggerMessage, myRemoteLocation);

messageText = “/recording/righthand/x/” + convertedRightHandPosition.x;
triggerMessage = new OscMessage(messageText);
oscP5.send(triggerMessage, myRemoteLocation);

// if we’re recording
// tell the record to capture this frame
if (recording)
{
recorder.recordFrame();
}
else if (playing)
{
// if we’re playing
// access the recorded joint position
PVector recordedLeftHandPosition = recorder.trackedJoints[0].getPosition().position;
PVector convertedRecordedLeftHandPosition = new PVector();
kinect.convertRealWorldToProjective(recordedLeftHandPosition, convertedRecordedLeftHandPosition);

PVector recordedRightHandPosition = recorder.trackedJoints[1].getPosition().position;
PVector convertedRecordedRightHandPosition = new PVector();
kinect.convertRealWorldToProjective(recordedRightHandPosition, convertedRecordedRightHandPosition);

// display the recorded joint position
pushMatrix();
stroke(0, 255, 0);
strokeWeight(30);
point(recordedLeftHandPosition.x, recordedLeftHandPosition.y, recordedLeftHandPosition.z);
stroke(255, 0, 0);
point(recordedRightHandPosition.x, recordedRightHandPosition.y, recordedRightHandPosition.z);
popMatrix();

String messageTextPlaying = “/playing/lefthand/x/” + convertedRecordedLeftHandPosition.x;
//println(messageText);
OscMessage triggerMessagePlaying = new OscMessage(messageTextPlaying);
oscP5.send(triggerMessagePlaying, myRemoteLocation);

messageTextPlaying = “/playing/righthand/x/” + convertedRecordedRightHandPosition.x;
triggerMessagePlaying = new OscMessage(messageTextPlaying);
oscP5.send(triggerMessagePlaying, myRemoteLocation);

// draw a line between the current position and the recorded one
// set its color based on the distance between the two
stroke(c, 255-c, 0);
strokeWeight(20);
line(leftHandPosition.x, leftHandPosition.y, leftHandPosition.z, recordedLeftHandPosition.x, recordedLeftHandPosition.y, recordedLeftHandPosition.z);
//stroke(c, 255-c, 0);
line(rightHandPosition.x, rightHandPosition.y, rightHandPosition.z, recordedRightHandPosition.x, recordedRightHandPosition.y, recordedRightHandPosition.z);
// calculate the vector between the current and recorded positions
// with vector subtraction
leftHandPosition.sub(recordedLeftHandPosition);
rightHandPosition.sub(recordedRightHandPosition);
// store the magnitude of that vector as the off-by distance
// for display
offByDistance = leftHandPosition.mag();
// tell the recorder to load up
// the next frame
recorder.nextFrame();
}

}
}
}// end draw

void keyPressed()
{
if (key == ‘ ‘)
{
recording = !recording;
playing = !playing;
}
}

// user-tracking callbacks!
void onNewUser(int userId)
{
println(“start pose detection”);
kinect.startPoseDetection(“Psi”, userId);
}

void onEndCalibration(int userId, boolean successful)
{
if (successful)
{
println(” User calibrated !!!”);
kinect.startTrackingSkeleton(userId);
recording = true;
}
else
{
println(” Failed to calibrate user !!!”);
kinect.startPoseDetection(“Psi”, userId);
}
}

void onStartPose(String pose, int userId)
{
println(“Started pose for user”);
kinect.stopPoseDetection(userId);
kinect.requestCalibrationSkeleton(userId, true);
}

class SkeletonRecorder {
private SimpleOpenNI context;
TrackedJoint[] trackedJoints;
int userID;
int currentFrame = 0;
int[] jointIDsToTrack;
int totalFrames;
boolean hasUser;

SkeletonRecorder(SimpleOpenNI context, int[] jointIDsToTrack) {
init(context, jointIDsToTrack);
}

SkeletonRecorder(SimpleOpenNI context, int jointIDToTrack) {
int[] joints = new int[1];
joints[0] = jointIDToTrack;
init(context, joints);
}

void init(SimpleOpenNI context, int[] jointIDsToTrack) {
this.context = context;
this.jointIDsToTrack = jointIDsToTrack;
}

void setUser(int userID) {
if (!hasUser) {
this.userID = userID;
hasUser = true;
trackedJoints = new TrackedJoint[jointIDsToTrack.length];

for (int i = 0; i < trackedJoints.length; i++) {
trackedJoints[i] = new TrackedJoint(this, context, userID, jointIDsToTrack[i]);
}
}
}

void recordFrame() {
for (int i = 0; i < trackedJoints.length; i++) {
trackedJoints[i].recordFrame();
}
totalFrames++;
}

void nextFrame() {
currentFrame++;
if (currentFrame == totalFrames) {
currentFrame = 0;
}
}
}

class TrackedJoint {
int jointID;
SimpleOpenNI context;
ArrayList frames;
int userID;
SkeletonRecorder recorder;

TrackedJoint(SkeletonRecorder recorder, SimpleOpenNI context, int userID, int jointID ) {
this.recorder = recorder;
this.context = context;
this.userID = userID;
this.jointID = jointID;

frames = new ArrayList();
}

JointPosition getPosition() {
return getPositionAtFrame(recorder.currentFrame);
}

JointPosition getPositionAtFrame(int frameNum) {
return (JointPosition) frames.get(frameNum);
}

void recordFrame() {
PVector position = new PVector();
float confidence = context.getJointPositionSkeleton(userID, jointID, position);
JointPosition frame = new JointPosition(position, confidence);
frames.add(frame);
}
}

class JointPosition{
PVector position;
float confidence;

JointPosition(PVector position, float confidence){
this.position = position;
this.confidence = confidence;
}
}

The main Pd patch, followed by the sub patch that handles the OSC messages.

Kinect Step Sequencer

Posted on by
Reply

Earlier this semester I made an air-drum kit, using a Kinect and Processing. I found the delay between the “striking” of the drums and the playback of the sound to be unacceptably long, but I still liked the idea of using 3-D hot spots to trigger audio in some way. At the same time I had been fumbling around trying to build a step sequencer in Pd. For the midterm I decided to combine the ideas. The Processing sketch shows the user a 16 step – 5 track grid of toggles, a start/stop button, and a set of virtual lights that indicate the current step. As toggles are triggered, OSC messages are sent to the Pd patch and routed to the corresponding toggles in the sequencer, which trigger audio samples at their designated point in the sequence. OSC is sent back to Processing to display the current step.

Click to download the full project including Processing sketch, Pd patch, and samples.

I think this project was successful to a degree, but it’s pushing the limits of what the Kinect is good at. Having so many toggles on the screen that need to be individually and deliberately triggered is a challenge because there is a high risk of inadvertently triggering a neighboring toggle. I tried to solve this problem by increasing the size of the sketch and spacing out the toggles more, but the Kinect has a relatively small peripheral range so it would lose track of my hand toward the edges. I was able to alleviate this problem to a degree by mandating that toggles would only be triggered if my hand was moving in a downward motion. Also, and maybe it’s just because I’m so used to pushing buttons and clicking a mouse, but I found the interaction to be physically unsatisfying. I think a more successful project would be one that translates the natural motion of the body into sound in a more fluid way, rather than relying on an awkward gesture. I initially intended this to be played with both hands, but found it worked better and was more natural to play with just one, so you’ll see some commented-out code referring to the left hand below. Ironically, since this project spawned from my failed air-drum set, I think I learned that using Pd to play the audio instead of Processing would actually enable me to create a good air-drum set. The communication between Processing and Pd over OSC is just about instantaneous and Pd is quick to play samples. I think it would work.

On the Pd side of things, the toughest part was figuring out the logic of the sequencer. After a lot of frustrating hours I was finally able to get one track going, but it was obviously inefficient and clunky… I had 16 objects that each loaded and played the same sample. I was happy to get that working, especially since it allowed me to start testing the communication back and forth between Pd and Processing, but I knew if I continued on that path I’d have a nightmare patch on my hands. Maybe I’m just a poor Googler, but I couldn’t find any sequencer examples out there that used toggles. Is the Pd community not interesting in programming hot beats? Luckily last week Hans posted a link to the mtl abstractions which included a couple of examples. I was able to modify one them to suit my needs. Turns out the spigot object is what I had been longing for all along. I ended up using a separate sequencer object for each track. The OSC messages get routed down the chain of sequencers until they find their home. Would’ve liked to have packed it all into one sequencer object but I just couldn’t figure it out.

//Processing Code

import java.awt.Rectangle;
import SimpleOpenNI.*;
import oscP5.*;
import netP5.*;

SimpleOpenNI kinect;
int zThreshold;

OscP5 oscP5;
NetAddress myRemoteLocation;

Rectangle startStop;
Boolean playing;
Boolean handInStartStop;
Boolean handPrevInStartStop;
color playingColor;
color notPlayingColor;

Rectangle[] triggers;
Boolean[] triggerStates;
Boolean[] handInTrigger;
Boolean[] handPrevInTrigger;
int tracks;
int steps;
int numTriggers;
int triggerWidth;
int rowIndent;
int rowHeight;
int topMargin;
int colWidth;
int colHeight;

color offColor;
color onColor;

Rectangle[] indicators;
Boolean[] indicatorStates;
int currentStep;
int previousStep;
color activeStepColor;
color inactiveStepColor;

float prevRightHandX;
float prevRightHandY;

void setup()
{
size(640, 480);
//size(1280, 960);
//size(960, 720);
//size(800, 600);
//size(720, 540);

kinect = new SimpleOpenNI(this);
kinect.enableDepth();
kinect.setMirror(true);
kinect.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);
zThreshold = 1600;

oscP5 = new OscP5(this, 12000);
myRemoteLocation = new NetAddress(“127.0.0.1″, 12001);

tracks = 5;
steps = 16;
numTriggers = tracks * steps;
triggerWidth = 16;
rowIndent = 22;
colWidth = triggerWidth + rowIndent;

topMargin = 28;
rowHeight = triggerWidth + topMargin;

startStop = new Rectangle(rowIndent + (colWidth * (steps – 1)), topMargin, triggerWidth, triggerWidth);
playing = false;
handInStartStop = false;
handPrevInStartStop = false;
playingColor = color(0, 0, 0, 255);
notPlayingColor = color(200, 200, 200, 200);

offColor = color(0, 0, 100, 150);
onColor = color(0, 200, 0, 255);

triggers = new Rectangle[numTriggers];
triggerStates = new Boolean[numTriggers];
handInTrigger = new Boolean[numTriggers];
handPrevInTrigger = new Boolean[numTriggers];
int triggerX = rowIndent;
int triggerY = topMargin + rowHeight;
for (int i = 0; i < numTriggers; i++)
{
triggers[i] = new Rectangle(triggerX, triggerY, triggerWidth, triggerWidth);
triggerX += colWidth;
if (triggerX > (width – colWidth))
{
triggerX = rowIndent;
triggerY += rowHeight;
}
triggerStates[i] = false;
handInTrigger[i] = false;
handPrevInTrigger[i] = false;
}

indicators = new Rectangle[steps];
indicatorStates = new Boolean[steps];
int indicatorRectX = rowIndent;
int indicatorRectY = topMargin + rowHeight + rowHeight * tracks;
for (int i = 0; i < steps; i++)
{
indicators[i] = new Rectangle(indicatorRectX, indicatorRectY, triggerWidth, triggerWidth);
indicatorRectX += colWidth;
indicatorStates[i] = false;
}
ellipseMode(CORNER);
activeStepColor = color(200, 0, 0);
inactiveStepColor = color(0, 0, 0, 100);

}// end setup

void draw()
{
background(220);

if (playing)
{
fill(playingColor);
}
else
{
fill(notPlayingColor);
}
rect(startStop.x, startStop.y, startStop.width, startStop.height);

for (int i = 0; i < numTriggers; i++)
{
if (triggerStates[i] == true)
{
fill(onColor);
}
else
{
fill(offColor);
}
rect(triggers[i].x, triggers[i].y, triggers[i].width, triggers[i].height);
}

int indicatorX = rowIndent;
int indicatorY = topMargin + rowHeight + rowHeight * tracks;
for (int i = 0; i < steps; i++)
{
if (indicatorStates[i] == true)
{
fill(activeStepColor);
}
else
{
fill(inactiveStepColor);
}
ellipseMode(CORNER);
ellipse(indicatorX, indicatorY, triggerWidth, triggerWidth);
indicatorX += colWidth;
}

kinect.update();
//image(kinect.depthImage(), 0, 0);

IntVector userList = new IntVector();
kinect.getUsers(userList);

if (userList.size() > 0)
{
int userId = userList.get(0);

if (kinect.isTrackingSkeleton(userId))
{
PVector leftHand = new PVector();
PVector rightHand = new PVector();

kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_LEFT_HAND, leftHand);
kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_HAND, rightHand);

ellipseMode(CENTER);
PVector convertedLeftHand = new PVector();
kinect.convertRealWorldToProjective(leftHand, convertedLeftHand);
fill(255, 0, 0);
//ellipse(convertedLeftHand.x, convertedLeftHand.y, 15, 15);

PVector convertedRightHand = new PVector();
kinect.convertRealWorldToProjective(rightHand, convertedRightHand);
convertedRightHand.x = lerp(prevRightHandX, convertedRightHand.x, .5);
convertedRightHand.y = lerp(prevRightHandY, convertedRightHand.y, .5);
ellipse(convertedRightHand.x, convertedRightHand.y, 15, 15);

println(“left hand z ” + convertedLeftHand.z);
println(“right hand z ” + convertedRightHand.z);

if (/*convertedLeftHand.z < zThreshold || */convertedRightHand.z < zThreshold && convertedRightHand.y > prevRightHandY)
{
for (int i = 0; i < numTriggers; i++)
{
if (/*triggers[i].contains(convertedLeftHand.x, convertedLeftHand.y) || */triggers[i].contains(convertedRightHand.x, convertedRightHand.y))
{
if (!handInTrigger[i] && !handPrevInTrigger[i])
{
triggerStates[i] = !triggerStates[i];
String messageText = “/” + i;
println(messageText);
OscMessage triggerMessage = new OscMessage(messageText);
oscP5.send(triggerMessage, myRemoteLocation);
println(“trigger ” + i + ” ” + triggerStates[i]);
handInTrigger[i] = true;
}
handPrevInTrigger[i] = true;
}
else
{
handInTrigger[i] = false;
handPrevInTrigger[i] = false;
}
}

if (startStop.contains(convertedRightHand.x, convertedRightHand.y))
{
if (!handInStartStop && !handPrevInStartStop)
{
playing = !playing;
OscMessage onoffMessage = new OscMessage(“/onoff”);
oscP5.send(onoffMessage, myRemoteLocation);
handInStartStop = true;
}
handPrevInStartStop = true;
}
else
{
handInStartStop = false;
handPrevInStartStop = false;
}
}

prevRightHandX = convertedRightHand.x;
prevRightHandY = convertedRightHand.y;

}

}

}// end draw

void oscEvent(OscMessage theOscMessage)
{
String beatMsg = theOscMessage.addrPattern();
int stringLength = beatMsg.length();
String beat = beatMsg.substring(1, stringLength);
println(beat);

currentStep = int(beat);
indicatorStates[currentStep] = true;
previousStep = currentStep – 1;
if (previousStep == -1)
{
previousStep = 15;
}
indicatorStates[previousStep] = false;

}// end oscEvent

// user-tracking callbacks!
void onNewUser(int userId) {
println(“start pose detection”);
kinect.startPoseDetection(“Psi”, userId);
}
void onEndCalibration(int userId, boolean successful) {
if (successful) {
println(” User calibrated !!!”);
kinect.startTrackingSkeleton(userId);
}
else {
println(” Failed to calibrate user !!!”);
kinect.startPoseDetection(“Psi”, userId);
}
}
void onStartPose(String pose, int userId) {
println(“Started pose for user”);
kinect.stopPoseDetection(userId);
kinect.requestCalibrationSkeleton(userId, true);
}

// mouse events for debugging
void mouseClicked()
{
if (startStop.contains(mouseX,mouseY))
{
playing = !playing;
OscMessage onoffMessage = new OscMessage(“/onoff”);
oscP5.send(onoffMessage, myRemoteLocation);
}

for (int i = 0; i < numTriggers; i++)
{
if (triggers[i].contains(mouseX,mouseY))
{
triggerStates[i] = !triggerStates[i];
String messageText = “/” + i;
println(messageText);
OscMessage triggerMessage = new OscMessage(messageText);
oscP5.send(triggerMessage, myRemoteLocation);
println(“trigger ” + i + ” ” + triggerStates[i]);
}
}

for (int i = 0; i < steps; i++)
{
if (indicators[i].contains(mouseX,mouseY))
{
indicatorStates[i] = !indicatorStates[i];
}
}
}// end mouseClicked

Main Pd Patch

Sequencer Abstraction

Kinect Drum Machine

Posted on by
Reply

For my Dataflow final project and Computational Cameras midterm I am building a step sequencer that will be gesture controlled using a Kinect, Processing, and Pd. Although I have mostly used Ableton to make music in recent years, back in the day I was a huge Fruity Loops fanatic and I’ve been itching to create my own sequencer for a while. This drum machine is the beginning of what hopefully will become a full scale production and performance tool.

This week I identified the various tasks that needed to be accomplished in order to make this work:

  • Create hot spots and have the Kinect recognize when a hand enters and exits a hot spot
  • Send OSC messages from Processing to Pd when a hot spot is triggered
  • Unpack OSC messages in Pd and trigger the corresponding note
  • Play the notes at the appropriate step in the sequence
  • Send OSC messages from Pd to Processing to reflect the current step in the sequence

So far I have been able to make it all work with just one track. Now I need to build out more tracks in Pd and construct the interface in Processing. Also, I feel like my Pd patch is going to be huge and clunky if I don’t figure out a way to encapsulate more stuff.

ADSR

Posted on by
Reply

This week we were assigned the task of using a line object to control sound or video. I decided to build upon my previous synthesizer. Last week I used a line just to control the attack of the note, using a knob on my MIDI controller to adjust on the fly. This week I decided to incorporate a complete ADSR envelope using 4 knobs to control all the parameters. I figured I could make a list of values separated by commas at the appropriate points and feed that into a vline object. Turns out commas are special in Pd so I couldn’t just send them into a list as symbols. After some frustrating hours I turned to Google and discovered there is a patch in the help browser, audio.examples/D02.adsr.pd, that could accomplish this task for me. Since my knobs send MIDI values between 0 and 127 I had to do a little math tweaking to convert to appropriate ranges. Looking at the logic of the ADSR patch I see that I didn’t actually need to pack the complete set of values to send to a vline. Instead, each step of the envelope is sent to a single line object with appropriate an appropriate delay between.

Click here to download the zip.

MIDI Controlled Synth

Posted on by
Reply

This week I decided to create a synthesizer using input from an external MIDI controller to shape the sound and trigger notes. I worked my way through the FLOSS audio tutorials and came up with a synth that combines two square wave oscillators, passes them through a square wave LFO, and finally through a bandpass filter. I used the the notein object to control the pitch and velocity of the first oscillator via my MIDI keyboard. I used the ctlin object to control the second oscillator, the LFO, and the filter via the knobs on my controller.

Download the patch.