Heather Velez

Comp Cameras: Punching Nun

Learning how to use skeletal tracking with the Kinect, I felt inspired by the punching nun puppets. It was simple enough to map the images of the nun’s head and boxing gloves to the appropriate joints. The difficulty came with how to scale the images based on the proximity of the user to the device e.g. smaller images farther away and larger images closer up. I also encountered a few issues with the background which disrupted the mapping of the images.

Processing Code:

import SimpleOpenNI.*;
SimpleOpenNI kinect; 

PImage r;
PImage l;
PImage nun;
//PImage bg; 

void setup() {
  size(640, 480);
  r = loadImage("rightglove.png");
  l = loadImage("leftglove.png");
  nun = loadImage("nunhead2.png");
  //bg = loadImage("Church1.jpg");
  //image(bg, 0,0); 

  kinect = new SimpleOpenNI(this);
  kinect.enableRGB();
  kinect.enableDepth(); 

  //turn on user tracking
  kinect.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);
}

void draw() {
  kinect.update();
  background(kinect.rgbImage()); 

  //make a vector of ints to store the list of users
  IntVector userList = new IntVector(); 

  //write the list of detected users
  //into our vector 

    kinect.getUsers(userList); 

  //if we found any users
  if (userList.size() > 0) {
    //get the first user
    int userId = userList.get(0); 

    //if we're successfully calibrated
    if (kinect.isTrackingSkeleton(userId)) {
      //mack a vector to store the left hand
      PVector rightHand = new PVector();
      PVector leftHand = new PVector();
      PVector head = new PVector(); 

      kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_LEFT_HAND, leftHand);
      kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_HAND, rightHand);
      kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_HEAD, head);

      PVector convertedRightHand = new PVector();
      PVector convertedLeftHand = new PVector();
      PVector convertedHead = new PVector();
      kinect.convertRealWorldToProjective(rightHand, convertedRightHand);
      kinect.convertRealWorldToProjective(leftHand, convertedLeftHand);
      kinect.convertRealWorldToProjective(head, convertedHead);
      //and display it
      fill(255, 0, 0); 

      imageMode(CENTER);
      image(nun, convertedHead.x, convertedHead.y);
      image(r, convertedRightHand.x, convertedRightHand.y);
      image(l, convertedLeftHand.x, convertedLeftHand.y); 

      // ellipse(convertedRightHand.x, convertedRightHand.y, ellipseSize, ellipseSize);
    }
  }
}

//user tracking callbacks
void onNewUser(int userId) {
  println("start pose detection");
  kinect.startPoseDetection("Psi", userId);
}

void onEndCalibration(int userId, boolean successful) {
  if (successful) {
    println(" User calibrated !!!");
    kinect.startTrackingSkeleton(userId);
  }
  else {
    println(" Failed to calibrate user !!!");
    kinect.startPoseDetection("Psi", userId);
  }
}

void onStartPose(String pose, int userId) {
  println("Started pose for user");
  kinect.stopPoseDetection(userId);
  kinect.requestCalibrationSkeleton(userId, true);
}

Final Presentation: Tantra Vision

I began working on my tantra vision by  replicating the one of the paintings  in Processing. 

 

 

 

 

 

 

 

 

 

 

I then chose the “historical” elements, both within the chosen painting and also from other paintings, that I wanted to include within the sketch and added those features.  For mockup purposes, I used the keyPressed function to call each element method so I could get a rough idea of what the experience would be like. This all came together pretty easily. The challenge then was how to connect the Processing sketch to Arduino. How could I take this interaction beyond keyPressed? As I mentioned earlier, I wanted to capture the breath of the user and have that control the sketch. But how could I do that? I did not want to encumber or constrict the user with a bulky interface (like a helmet) or a chest strap. I wanted the physical interface to be as non-invasive as possible. So I set out researching several things: a C02 sensor, electret microphone, and  a heart rate monitor.

Inevitably, all of the parts that I ordered from Sparkfun and Parallax took an obscene amount of time to arrive. Given the circumstances, I quickly ruled out the heart rate monitor and was able to borrow an electret microphone (from Scott, thank you) and a CO2 sensor (from Tom, thank you) to start experimenting. I had to figure out what type of values I could get out of these sensors and if they would be useful for me in this project. When I started mapping the values, I found that the range was very small. I realized that it would take a lot more work (a visit to Eric Rosenthal to learn about inverting the sound wave) and may not be the best way forward.

 

 

 

 

 

 

 

 

So I circled back to the breathing idea – if I couldn’t capture breath, maybe I could animate the sketch in a way to appear as if it was breathing. I could use variable resistors, a set of FSRs, to translate the pressure exerted by the user onto the sketch. But I still had a problem: where would I put the FSRs?

Since tantra paintings are used for meditation, I was perplexed by the idea of introducing some sort of physical element. I thought something small, intimate, might be appropriate. Perhaps a cup or a bell? Maybe even a singing bowl? It wouldn’t be the most visually appealing though to have a cup with FSRs mounted on it and wires dangling down. That’s when I thought about an instrument, a didgeridoo, to be specific.

 

 

 

 

 

 

 

 

I set about fabricating my five-foot long didgeridoo (out of PVC pipe and colorful tape) and mounted the FSRs like keys on the outside of the pipe. When I finally hooked it up to the Arduino and ran the sketch, I discovered an immediate problem. Scale. A five foot didgeridoo and a sketch running on a computer screen are not the most compatible. And if I was attempting to create some sort of intimate, personal experience having a physical distance so great between the user and the visualization just wouldn’t work. Back to the drawing board, I go.

 

 

 

 

 

 

 

 

Several residents suggested that I consider using the Kinect which now has the ability to measure the rise and fall of a user’s chest. So that’s my next step. Start playing around with the Kinect, see what I can get out of it and run with it. Additionally, I would like to make more of these paintings so that there’s are many options for the user to explore.

 

Final Project: Tantra Drawing Visualization

My final project is inspired by the Tantric paintings collected by a French poet, Franck André Jamme, in Rajasthan, India. These paintings serve as an aid to meditative practice and use simple, conventional symbols  to stimulate specific mental and spiritual experiences. Most of the drawings in his collection have only one or two shapes, with no ornamentation and are monochromatic. What I find so compelling beyond the purpose of the paintings (as meditative aids) is the history of each. These paintings are handmade, copied, and passed down through generations. There are imperfections in the paper, tape and pin marks, water stains, ink transfers, and tears. There is beauty in that layering. The life of the painting grows and takes on a new meaning / multiples meanings in these incidental marks / what remains on the page.

In Processing, I would like to recreate one of the images in its original state – just the shape or shapes with none of the historicity. Then slowly through user interaction illustrate (my interpretation) of the evolution of the painting. Starting with perhaps the fading around the edges, then maybe a tear, some tape parks, errant text. The raw nature of the “original” painting would be exposed as the narrative develops (its authenticity erased).

The problem is what type of user interaction?  I had originally thought about creating a helmet the user would wear to view and explore the image. But that seemed clunky and more of an impediment to meditation. When I posed this question in my ICM class, I received some valuable feedback. One of the central principles of meditation is breathing so why not utilize that for the user interaction?

In my wildest dreams, I would love to create a screen that is sensitive to breath and responds to each exhalation. I’m not sure how possible that is but I know there are other ways to measure breath. I could use a microphone to pick up the sound of breathing or a chest strap with a flex sensor that would pick up the contraction of each breath.

Alternatively, I could:

• Measure heart rate and as it slows reveal more of the history of the image
• Use facial recognition to sense a user and then start sequence after they’ve inspected the image for a few seconds
• Use eye tracking to reveal different aspects of the history as the user explores the image

Building a Heart Factory

For my ICM midterm, I decided to design a “heart factory.” I envisioned a factory floor background with a conveyor belt in the foreground; a large spout (like a sewage pipe) placed either on the side or at the top of the sketch and hearts being pumped out from the spout. The hearts would cascade onto the moving conveyor belt one by one and eventually leave the sketch screen.

My first task was how to make the conveyor belt. How would I create the illusion of movement? How would I create depth so it didn’t appear so flat? If I created an array of dotted lines that moved along the x-axis, that might work. I could have cogs rotating on either end of the conveyor belt as an additional visual aid. I ended up getting some (a lot) of help from another classmate doing a similar project.

Once the conveyor belt was up and running, the next step was to create the hearts. I immediately thought of the candy hearts so popular on Valentine’s Day. Could I draw the heart? Or load an image? If I decided to draw the heart, I would also want to include text. Every year, the text printed on these little hearts just keeps getting better! “Txt Me,” “UR A QT,” “LETS READ,” are just a few examples.  They’re really keeping up with the times here – got their finger on the pulse. Anyway, I digress, so ideally I would create an array of hearts with various texts that would either fall in a group out of the spout or individually and onto the conveyor belt.  It seemed easier to guide an individual heart as it fell from the spout, onto the belt, and then off screen. As the heart falls down the y-axis, once it hits a certain Y point then there would be a momentary stop before it started moving along the x-axis. So my next task was defining the parameters for this l-shaped movement.

My next steps:

• If I decide to use images for the hearts, create an array of hearts & make the move.
• If I decide to go with the heart shape I drew, create an array of heart shapes & make the move.
• Secondly, for the heart shapes incorporate the text as a random array so that there’s some variation with the hearts. I was able create an array of strings for text but I haven’t been able to match the object to the string.

 

Modularity

I find the word “modularity” a little awkward. It doesn’t roll off my tongue and I need to visually map all of the syllables as I say it (which isn’t a particularly desirable effect). The concept of modularity itself, fortunately, is much easier for me to understand. Simply put, it’s a set or system of interchangeable parts. And it manifests across a number of disciplines and industries including computer programming, math, architecture, and product design. Legos is a great example. A standard set of Lego blocks provides a seemingly limitless set of combinations to the user. I like to make Space Invaders shapes with my Legos but obviously I’m not limited to just that. One of my classmates shared the work of a phenomenal artist that works in Legos, Nathan Sawaya.

I also think that widgets and plug-ins fit the “modular” bill. Aidan Feldman created mustachify.me, a widget that uses the face.com API to place a moustache on any image (as shown above). Just add your image URL to the end of the address “http://mustachify.me/?src=YOUR_IMAGE_URL” and voila!

Self-Portrait In Processing

For my first assignment in Introduction to Computational Media, I drew a self-portrait in Processing. I used code from Heather Dewey Hadborg’s interpretation of Sol LeWitt’sLine Drawing 105 to create the background. After many failed attempts to create a curve, I decided to try my hand at creating a bezier curve. I found the bezier curze much easier to grasp conceptually and aesthetically pleasing.

In my first version, I used arcs to create the half-elliptical shapes for my hair and eyeglasses. I wasn’t very satisfied with the visual. I wanted to create a self-portrait with a little more fluidity. It felt a little too flat to me.

 
 

For my second version, I used Illustrator. I didn’t mind my bumpy, unpolished lines. This felt a lot closer to what I envisioned. Then the Illustrator to Processing script didn’t work. Not equipped to clean up the Processing script, I started a new sketch (my final version)!

Check out the code here.