Time-Based Sculpture – Final Update

After presenting my progress in Nature of Code last week, I continued experimenting with the Trace Modeler software from Interactive Fabrication. The software uses OpenCV to do blob detection, and draws a point cloud around the perimeter of the blob in 3D space over time. There are a lot of useful features you can set, like the distance between the planes in Z-space, the smoothness of the surface mesh, and the capture rate controlling the amount of frames it plots.

In order to get the Trace Modeler software working correctly, you need to have OpenFrameworks running on your computer. OpenFrameworks is an opensource library for developing C++ applications. You also need a development environment like XCode installed. As it says in the “Read Me” text, the Trace Modeler software requires the following addons – ofxVectorMath, ofxSTL, ofxDelaunay, ofxMSAInteractiveObject, ofxXmlSettings, ofxSimpleGuiToo, and ofxOpenCv. Download these additional libraries and place them in the add-ons folder inside the OpenFrameworks folder in your Documents. It should probably go without saying at this point, but I’m on Mac OSX Snow Leopard. In order for the program to export the STL files, you need to change the file path in the traceModel.cpp file to correspond with the bin folder of the the application on your computer. I didn’t specify any path, just stlExporter.saveModel(“testBin.stl”); and that seemed to work fine. When you press the Capture button it saves an .STL file to the folder.

Trace Modeler OpenFrameworks Software

Test to see if the application is running by pressing the “Build and Run” green button at the top of the XCode window. Once it’s running, you can see the camera view in the upper left of the screen, and the contrasted camera view doing the blob tracking in the upper right. The red material being drawn to the main screen is the 3D model being written. It helps to have a contrasted background in order for the blob tracking to work to the best of its ability. If the object or person being tracked is lighter, then a black background works well, and if the object is darker a light background works. Lighting also helps darker objects contrast against a black backdrop, which seems to absorb shadows. Once the background, camera view, and lighting are set, make sure nothing is in the camera view and press the “Learn Background” button on the application, which sets the reference frame. Then start experimenting with the settings for Threshold (sets the threshold for the blob tracking), Z-Index (sets the distance in 3D space between the perimeters), and Capture Rate (the time in milliseconds between each camera draw). Smoothing also affects the smoothness of the polygons. When you’re satisfied with the shape (or just want to see what it looks like), press “Export Model,” or “Clear Shape” if you want to start over once the settings are how you want them.

Next I opened up the STL file in MeshLab to see how the model looked in 3D. Since the mesh was already made in OpenFrameworks, I only had to make surfaces for the two ends. This proved to be the trickiest part. Many thanks to Miguel Bermudez for his expertise with 3d-modeling.

[Show as slideshow]

Time-Based Sculpture – Progress Report

For my Nature of Code final, I decided to pursue the gestural approach to create a time-based sculpture by experimenting with the Kinect and its point cloud capabilities for 3D printing. After doing some initial research into best options for this approach, I was pointed to work done by students in the 3D Sensing class taught by Zach Lieberman and Kyle McDonald this spring. Kyle has some code posted on github that generates a .ply file of a point cloud, which you can bring into Meshlab (open source 3D modeling software) to generate a surface mesh for printing. You can download the bundle for the whole class. The code I used is from week 6, called PlyExporter.

Now, I was able to get the code running in OpenFrameworks, and did get some .PLY files into MeshLab. However, since I am interested in capturing a gesture over time, I needed to change the code in order to capture the points from multiple frames in a short span of time. Just pressing the record button multiple times didn’t quite capture the fluidity I was looking for. Since I’ve never coded in OpenFrameworks before, I decided to see if I could something like this working in Processing. After consulting the internet for a bit, I found a tutorial that claimed to do this very thing.

The example code available on the site uses an older version of Shiffman’s kinect library for processing, so unfortunately I wasn’t able to get it running until Shiffman was kind enough to look it over and update the syntax. The original code accessed the Depth Image and stepped through the pixels. The brighter the pixel value, the “closer” the pixel is in 3D space. The updated code is adapted from the Point Cloud example, and accesses the depth value array instead of scaling the brightness of the Depth Image values to render the point cloud into 3d space. The point cloud example makes use of the DepthToWorld() function, that translates what the Kinect sees as values from 0 to 2048 into meters.

Here is the code Shiffman updated which exports an .OBJ file of the point cloud in view after pressing the ‘R’ key. You will need to download the openkinect library, the PeasyCam library, and the SuperCAD library.

/*
 Genevieve Hoffman - Point Cloud to OBJ file, Nature of Code Final Project 2011
 
 Based on Simple Kinect point-cloud demo v. 0.2, by Henry Palonen <h@yty.net>
 
 Using Daniel Shiffman’s great processing-library for Kinect and Point Cloud Example:
 
 http://www.shiffman.net/p5/kinect/
 
 Based on Kyle McDonalds Structure Light scanner:
 
 http://www.openprocessing.org/visuals/?visualID=1014
 
 Using also SuperCAD for outputting the .obj – files:
 http://labelle.spacekit.ca/supercad/
 
 History
 ——-
 17.11.2010 – 0.1 – First version, simple point-cloud working
 18.11.2010 – 0.2 – Output to .obj for importing to Blender,
 gray-color for distance and small lines as output
 
 */
 
import peasy.*;
import org.openkinect.*;
import org.openkinect.processing.*;
import superCAD.*;
 
boolean record = false;
 
float zscale = 3;
float zskew = 100;
 
int inputWidth = 640;
int inputHeight = 480;
 
PeasyCam cam;
 
PVector[][] gray = new PVector[inputHeight][inputWidth];
 
// We'll use a lookup table so that we don't have to repeat the math over and over
float[] depthLookUp = new float[2048];
 
 
PImage depth;
 
static final int gray(color value) {
  return max((value >> 16) & 0xff, (value >> 8 ) & 0xff, value & 0xff);
}
 
Kinect kinect;
 
 
void setup() {
  size(inputWidth, inputHeight, P3D);
  background(0);
  cam = new PeasyCam(this, width*2.5);
  kinect = new Kinect(this);
  kinect.start();
  kinect.enableDepth(true);
  //depth = createImage(640, 480, RGB);
  stroke(255);
  // Lookup table for all possible depth values (0 - 2047)
  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
  }
}
 
int counter = 0;
 
void draw () {
  background(0);
  if (record == true) {
    beginRaw("superCAD.ObjFile", "Rhazes_" + counter + ".obj"); //Start recording to the file
  }
 
  //depth.pixels = NativeKinect.getDepthMap();
  //depth.updatePixels();
  //depth = kinect.getDepthImage();
  int[] depth = kinect.getRawDepth();
 
  for (int y = 0; y < inputHeight; y++) {
    for (int x = 0; x < inputWidth; x++) {
      int offset = x+y*inputWidth;
      // Convert kinect data to world xyz coordinate
      int rawDepth = depth[offset];
      PVector v = depthToWorld(x,y,rawDepth);
      gray[y][x] = v;
    }
  }
 
 
  float scaling = 200;
 
  // Kyle McDonald’s original source used here
  translate(-inputWidth / 2, -inputHeight / 2);
  int step = 2;
  for (int y = step; y < inputHeight; y += step) {
    float planephase = 0.5 - (y-(inputHeight / 2)) / zskew;
    for (int x = step; x < inputWidth; x += step)
    {
      stroke(255);
      //point(x, y, (gray[y][x] - planephase) * zscale);
 
      PVector v = PVector.mult(gray[y][x],200);
 
      if (v.z > 100 && v.z < 200) {
        line(v.x,v.y,v.z,v.x,v.y,v.z);
        //line(gray[y][x].x,gray[y][x].y, gray[y][x].z,gray[y][x].x,gray[y][x].y, gray[y][x].z);
      }
    }
  }
 
  if (record == true) {
    endRaw();
    record = false; // Stop recording to the file
  }
}
 
void keyPressed() {
  if (key == 'R' || key == 'r') {
    // Press R to save the file
    record = true;
    counter++;  //Thanks to Dr. Rhazes Spell for putting the counter in adding functionality.
  }
}
 
// These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
float rawDepthToMeters(int depthValue) {
  if (depthValue < 2047) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}
 
PVector depthToWorld(int x, int y, int depthValue) {
 
  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;
 
  PVector result = new PVector();
  double depth =  depthLookUp[depthValue];//rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}

After you export the .OBJ file, the next step is to bring it into MeshLab to clean up the points and generate a surface mesh, which you need in order to print 3D. I recommend checking out Kyle’s useful tutorial that explains some of the processes you need to go through in MeshLab to take generate a surface mesh for the exported point cloud. If you want to give this method a try, I highly recommend watching his tutorial before trying to figure out MeshLab on your own.

Since my goal was to capture a three-dimensional gesture over time, I ended up changing the code posted above a bit in order to export multiple frames of point clouds. I’m happy to post the code, but I’m not sure it’s quite at the state to be most useful, since I haven’t been able to implement controls for how many points to add to the arraylist which gets exported.

The code exports many .OBJ files, about the same as the amount of frames the code has been running for. Next I opened up MeshLab and imported the .OBJ files to see what they looked like and how they might fit together. MeshLab, though it is freeware, has a ton of options about how to create meshes from point clouds. Again, I went off from Kyle’s tutorial, but the basic workflow for joining multiple point clouds is get rid of all extra points with the select and delete functions. Once you have the point cloud looking for the most part how you want it, calculate the normals for the points. By default, the normals should draw behind the points (this wasn’t quite true for me since I had so many points). Next you recalculate the normals, the generate a mesh. There are a ton of ways to do this, and I spent a lot of time trying to figure out which method would yield the results that I wanted. Kyle’s workflow didn’t quite end up working for me since I had so many interior points built up, so I experimented a lot with other methods of building surface meshes.

Here are some screenshots of my workflow and process. The point cloud captures two hands coming together in a sort of prayer-like gesture.

Point Cloud brought into MeshLab

Point Cloud Downsampled with Poisson-Disk Sampling

Side View of Point Cloud in Mesh Lab

Poisson Surface Reconstruction - following Kyle McDonald's Tutorial, getting weird results

Mesh View after VCG Surface Reconstruction - still artifacty

Textured Mesh View after Alpha Complex Reconstruction - very polygonal

Surface Mesh View after Marching Cubes (APSS) Reconstruction - getting better

Textured Mesh View after Marching Cubes (APSS) Reconstruction

Textured Mesh View after Marching Cubes (APSS) Reconstruction - adjusting parameters

Surface Mesh View after Marching Cubes (APSS) Surface Reconstruction - almost there

Textured Mesh View after Marching Cubes (APSS) Surface Reconstruction - better parameters

Basically, it comes down to being able to use a 3D modeling program to create the best surface mesh you can from the point cloud. My time-based point cloud and all its points inside the object (which I was going to start erasing in MeshLab to see how that would work but didn’t quite have time) is confusing for the 3D modeling software because it wants to draw polygons connection all those points as well. I had to choose between spending the time to do that in the modeling software, or figuring out a smarter way to capture a point cloud on the processing side. Since just capturing a point cloud gives me too many points, I realized that a combination of a perimeter over time, with a point cloud for the “end point” might be the best method. OpenCV blob tracking will do this pretty easily. I was going to start coding this, and then I found this project on the internet.

The people who also brought you the Kinect puzzle pieces wrote software to trace the perimeter of an object and draw it over time in 3D space. They have also kindly posted their source code, which is in OpenFrameworks, on the website. At this point I have the code running in OpenFrameworks and am doing tests to see what I can capture, and how best to do this.

Hopefully I will have some models to test print in the next few days, in the meantime, here is a video of the software in action:

Special thanks to Corrie Van Sice, Ezer Lichtenstein, Greg Borenstein, Nikolas Psaroudakis, and Dan Shiffman for their help and advice throughout this process.

Nature of Code Final Proposal

For my final project in Nature of Code, I would like to try and make a time-based sculpture. Many time-based sculptures involve aspects of traditional time-based media, like video or sound. I want to try and make a static sculpture that embodies a time-based process. There are two directions I am thinking of going in to achieve this.

1. Generative

I want to experiment with a few generative algorithms and draw their process of evolution out in 3D space. Possible algorithmic models I’m looking at are Cellular Automata, Reaction-Diffusion (also known as the Gray Scott model), and 3D fractals.

In my research, I came across the work of computational architect Michael Hansmeyer. I am especially interested in his Platonic Solids series, which takes a traditional Platonic Solid, and performs an iterative process subdividing its faces into more faces, essentially generating a three-dimensional fractal. This process, like fractal patterns in nature, generates amazing complexity from a simple process. He varies the value which is performing the operation, which produces more variation over time.

2. Gestural

Another direction I am considering is trying to make a sculpture that captures a moving gesture. I am interested in trying to make the ephemeral tangible, or more permanent. I am also interested to see what capturing a motion over time would look like when it’s layered over itself. My instinct is that it would turn something recognizable, like a hand, into something more abstract.

Working from the Point Cloud example for the Kinect, my goal would be to visualize and save the data points from multiple frames. I would most likely export these into a 3D modeling program to see what it would like, and perform any smoothing. I see this project as an experimental framework, since I’m not completely sure what sorts of gestures are worth “saving,” and what types of abstractions might appear.

Here is video documenting some works by the artist Koki Tanaka. Since the late 60s, video has been the primary form to document time-based performance sculptures. The result of these works is a record of the action, not the goal of the works, and most often secondary to the process itself.

Here is another video piece (sorry about the ad), focused on simple hand gestures. This might be a direction I experiment with once the framework is functional. Hands have always fascinated me in their complexity and utility. It would be interesting to try some video experiments with hands holding various tools – pencils, water glasses, knitting needles.

Gesture-Sculpture by bernardperroud

3. Gesturally generative

A third direction I am considering is to control a 3D particle system, or flow field, with physical gestures. I’m not sure if the final output would be sculptural, but I do think it would be an interesting experiment to affect a three-dimensional environment.

Here is a video of a 3D flow field by Reza Ali.

Apparently this direction is being taken by research done at Microsoft, surprise surprise.

Kinect and 3D Printing

I am trying to mull over how to generate 3D sculptures generated from kinect depth data. I want to start working from Dan Shiffman’s Point Cloud example, and figure out how to record depth data for a specific gesture over time. Then I want to collapse the data into one instance, and see what abstractions appear once it’s printed 3D.

Here are some links and video of what people are doing with the Kinect and 3D printing at this point.

This happened at the Tangible, Embedded, and Embodied Interaction conference. How do I get to go to this next year? via Joystiq.

Here the Fabricate Yourself team describes their process in more detail.

They seem to be outputting STL files. I will have to research those a bit more, as well as what 3D printer I’m going to use. I hear they’ve been doing a lot of it at MakerBot so I may try and check that out.

This post by FaceCube seems to be reacting to the fact that Fabricate Yourself didn’t publish their methods. Here is a tutorial with links to a library he used (made?) to render out the 3D. I’m just not sure which language he’s using.

These tutorials will also probably prove helpful. It seems like Jose Sanchez at genware.org has already figured out how to export the data from a point cloud. Here is also a page of links on algorithms for programming natural behaviors.

Midterm v2.0 – Edge Particles with Steering Forces

I decided to try and implement the OpenCV – Kinect blob detection sketch with steering forces governing the particle / edge attraction instead of the attractor / repulsion gravity force method that wasn’t quite giving me what I wanted in my previous iteration. At this point I’ve succeeded in populating the edges of the OpenCV blobs with target PVectors that the particles are attracted to. I’ve only really done a mashup using Dan Shiffman’s steering example and combining it with the Kinect and OpenCV blob edge detection code I was using before.

Here are some screen videos of how it’s behaving right now:

And here it is with the OpenCV image visible to give an idea of how it’s following the body:

The next step I’d like to take is bring the depth information into the particles as a z-vector that would affect their speed traveling to the edges of the body. I haven’t quite been able to make this work yet. I’d also like to add a repulsion force between the particles so they’re not so closely spaced. Although the effect I have going now isn’t what I had intending, it is sort of interesting, so I might keep going in this direction.

Midterm v1.0 – Edge Particles with Attractor / Repulsion forces

Here is the progress of my midterm project I worked on for Computational Cameras and Nature of Code. I wanted to create a system where a viewer would see themselves abstractly made up of particle or molecule type objects. This was inspired by a lifesize graphite drawing I saw a while back by Nathaniel Price.

This is the only image I could find of the piece

In order to make particle objects populate around a person’s body, I decided to use the Kinect camera to set a threshold. Using a threshold, I made a PImage and drew black pixels for anything in front of the threshold and white pixels for anything behind it. OpenCV then did blob detection on the image and returned an array of blobs. Next I called the OpenCV “points” function to step through the array of pixels that made up the perimeter of each blob. I divided this amount by the number of attractor objects I wanted the system to have, which gave me a way to disperse them regularly around the blob using this interval. Once I drew the attractor objects, the particle objects had an imbedded force calculation so that they would determine which attractor they were closest to, and become attracted to that attractor.

Here are a few screen videos of the program running. The first shows you how it’s running as of now (a bit slower due to video capture). Even without capturing video at the same time it’s still fairly slow because OpenCV can be slow, and I’m also constantly updating the attractor location, which in turn means every particle has to recalculate which attractor to be attracted to. There is a better way to do this, but my program kept breaking when I tried to change the for loop so that’s something I can revisit if I decide to keep moving forward with this direction.

The next video shows you the same program, but is drawing the attractor objects as gray dots to give an idea of how they’re finding the edges of the blobs, as well as updating every 15 frames.

The last video also shows the Kinect / OpenCV Image that is having the blob detection run on it:

I’ve been having some problems trying to figure out how to make the attractors populate in the center of the body, not just around the perimeter. It’s a matter of finding the center point, then radiating out I believe, but my code keeps breaking when I try to implement that. After speaking with Dan Shiffman, he encouraged me to leave the attractor / repulsion gravity force model I’d been using, and try to see if I could give the particles a steering force instead, so that they’d “arrive” at target points dispersed around the perimeter of the body.

Midterm Progress – OpenCV Blob Detection with Kinect

For the computer vision portion of my midterm project for Comp Cameras and Nature of Code, I decided to use the Kinect camera to separate people from a background, and then run OpenCV Blob Detection to find their contours. Eventually, I am going to populate the outlines of peoples’ bodies with Attractor objects, so that Particle objects start to delineate their bodies. For more about the concept and goals of the piece, please read more here.

In order to see if I could initiate attractor objects around the edge of a person, my first step was to start with a static image. I used the Blob Detection library in Processing to trace the edge around a silhouette of a woman. Next I asked for the array of edge vertices that made up the outline, divided the number of edge vertices by the number of attractor objects I wanted to draw to get the spacing interval, and stepped through the edge vertices by that amount each time.

Drawing the edge in green, the bounding box in red, and the attractor objects in gray

It was definitely tricky trying to figure out the correct way to write the ‘for loop’ for stepping through all the vertices, and writing the ellipse at specific intervals. For a long time I could only see one ellipse, since I was always writing it at the same place. After a bit of guess and checking, and then trying to think logically, I finally got what I wanted.

The next step was to put some particle objects into the system as well and see how they were attracted to the outline of the figure, to see if I was going in the right direction for what I want the behavior to do.

Particles and Attractor location based on Edge Detection

Not drawing the image, but doing the edge detection, gives you a little more of an idea how it looks with only the particles visible. The attractors are drawn lightly in gray but should be turned off.

Without drawing the silhouette image or edge detection

After I got my code working on a static image, I decided to try for a moving one. I also wanted to use the Kinect as my camera input since the depth map makes it so simple to separate a person out from a background, and later I want to see if I can use the z axis data to affect the particles in some way as well. I decided to switch blob detection libraries and go on to OpenCV, since it seems to be more powerful and has so many other capabilities in addition to blob tracking. I owe a big thanks to Craig Kapp and Saul Kessler for their help getting me up and running with the Kinect talking to OpenCV.

One issue I spent a while trying to figure out was why the blob detection in OpenCV didn’t seem to be working. Starting with Dan Shiffman’s RGB Depth Test, I was taking depth map information, and based on a threshold, telling the computer to display that information as a PImage, with the pixels in back of the threshold white, and the ones in front gray. Next I wanted OpenCV to do blob detection on the PImage, and then populate the edges with attractor objects. For the longest time I was only able to draw attractors in the four corners of the screen, as you can see below.

Turns out OpenCV can only “see” in black and white, namely it can only do blob detection on images whose pixels are entirely 0 or 255. My problem was that I was drawing the information in front of the threshold as gray, which wasn’t appearing at all. By default, OpenCV decided I wanted to do blob detection around the entire PImage itself, since all it could find was a large rectangle of white pixels. Once I changed that it I was able to draw ellipses around the edge of the moving figure.

Not pretty but much more functional!

Midterm Proposal – A particle based installation

A while back I had the chance to visit the diRosa Preserve, a museum of sorts in Napa, CA showcasing an amazing collection of contemporary art collected by Rene and Veronica di Rosa. I strongly encourage anyone visiting wine country to check it out. It truly is a wonderful place. But I digress.

I saw a lot of wonderful work that day, including a beautiful installation of Chartres Bleu by Paul Kos (it’s worth visiting just to see this piece), but a piece that keeps recurring in my mind is by an artist named Nathaniel Price, called “Another Matter XIII,” made in 2002. In the lifesize drawing, circular marks abstractly delineate the figure of a person hung upside down. There are surprisingly few resources about Price’s work on the internet, but here is a link to a Kenneth Baker review of the show where I believe the diRosas bought the drawing. I remember the piece as graphite on paper, but it seems that he used hot circular objects to burn into the paper.

This is the only image I could find of the piece

For my midterm, I propose to make an installation where person’s body is abstractly formed from particle objects. When no one is in front of the installation, there will be particle objects moving around the screen. When someone approaches, attractor objects will initiate around their outline, so that the particles will slowly start to move towards their figure (the attractor objects won’t be drawn, nor will the person’s video image be drawn, so the only way they will see themselves “represented” is from the particles moving towards them. In terms of quality, I want the installation to respond to stillness, where the particles will move fairly slowly, and only settle when the person is standing still.

In terms of what the particles look like, I am going to start with drawing them as outlined circles, but if I have time I would like to experiment with importing textures so that the particles appear more “hand drawn.”

Kinect + Particle Systems

I decided to try and see how I could implement some of the sketches I’ve been doing for Nature of Code with the tracking capabilities of the Kinect. At this point it is still pretty rough, but basically I’m adding a Particle System to follow the Average Point Tracking sketch instead of an ellipse. I left the IR image of my hand and body in front of the threshold to show what is happening, which is that the Particle System is following the average point of whatever region is in front of the threshold.

My attempts to combine the particle system with sketches that track one spot haven’t been successful as of yet, but I will keep working on it this week since I would like to continue developing this project for the midterm.