After presenting my progress in Nature of Code last week, I continued experimenting with the Trace Modeler software from Interactive Fabrication. The software uses OpenCV to do blob detection, and draws a point cloud around the perimeter of the blob in 3D space over time. There are a lot of useful features you can set, like the distance between the planes in Z-space, the smoothness of the surface mesh, and the capture rate controlling the amount of frames it plots.
In order to get the Trace Modeler software working correctly, you need to have OpenFrameworks running on your computer. OpenFrameworks is an opensource library for developing C++ applications. You also need a development environment like XCode installed. As it says in the “Read Me” text, the Trace Modeler software requires the following addons – ofxVectorMath, ofxSTL, ofxDelaunay, ofxMSAInteractiveObject, ofxXmlSettings, ofxSimpleGuiToo, and ofxOpenCv. Download these additional libraries and place them in the add-ons folder inside the OpenFrameworks folder in your Documents. It should probably go without saying at this point, but I’m on Mac OSX Snow Leopard. In order for the program to export the STL files, you need to change the file path in the traceModel.cpp file to correspond with the bin folder of the the application on your computer. I didn’t specify any path, just stlExporter.saveModel(“testBin.stl”); and that seemed to work fine. When you press the Capture button it saves an .STL file to the folder.
Trace Modeler OpenFrameworks Software
Test to see if the application is running by pressing the “Build and Run” green button at the top of the XCode window. Once it’s running, you can see the camera view in the upper left of the screen, and the contrasted camera view doing the blob tracking in the upper right. The red material being drawn to the main screen is the 3D model being written. It helps to have a contrasted background in order for the blob tracking to work to the best of its ability. If the object or person being tracked is lighter, then a black background works well, and if the object is darker a light background works. Lighting also helps darker objects contrast against a black backdrop, which seems to absorb shadows. Once the background, camera view, and lighting are set, make sure nothing is in the camera view and press the “Learn Background” button on the application, which sets the reference frame. Then start experimenting with the settings for Threshold (sets the threshold for the blob tracking), Z-Index (sets the distance in 3D space between the perimeters), and Capture Rate (the time in milliseconds between each camera draw). Smoothing also affects the smoothness of the polygons. When you’re satisfied with the shape (or just want to see what it looks like), press “Export Model,” or “Clear Shape” if you want to start over once the settings are how you want them.
Next I opened up the STL file in MeshLab to see how the model looked in 3D. Since the mesh was already made in OpenFrameworks, I only had to make surfaces for the two ends. This proved to be the trickiest part. Many thanks to Miguel Bermudez for his expertise with 3d-modeling.
For my Computational Cameras final, I set out to build an Android application that would serve as the sensor device for the Darkness Map project. To my delight, I was actually able to get something working via Android. As of now, I have an app running on my phone which analyzes the average brightness of each frame of the camera preview. It also keeps track of the time stamp, longitude, latitude, and accuracy of the GPS. These values get stored up in an array and sent to a server every 64 frames (so as not to overload the server or slow the app down by constantly sending values over the network). You can find the data online at itp.nyu.edu/~gh726/darknessmap/rawdata/longdata.txt.
One of the hurdles that I encountered is that most Android phones don’t have nice enough cameras that can disable the auto-exposure function. In order for the Darkness Map to have data that’s somewhat accurate, I need to disable the auto-adjustments so that the camera doesn’t adjust the exposure down when it sees a very bright area, or adjust up when it’s in darkness. After searching around the internet for a long time, I realized that this was going to be hard to control via hardware, mainly because most camera phones are built so that they have the opposite capability, to deliver the best picture quality in a variety of lighting conditions. Shawn Van Every, who teaches Mobile Media, was kind enough to point me to the getExposureCompensation() function, which returns the value that the camera exposure is adjusting for. If this function returns 0, then camera exposure adjustment is not supported (which is ideal for my purposes). However, my phone’s camera adjusts between a range of -4 to 4.
So, I haven’t implemented this functionality yet, but I believe that keeping track of the amount that the phone is adjusting the exposure will allow me to weight the brightness readings by taking into account the degree of exposure adjustment the camera made. For example, if the camera returns a midrange brightness reading of 100, but the Exposure Compensation was a full 4, then I’m pretty sure it’s actually a lot darker than a value of 100 would indicate. I need to do a lot more testing and play around with various equations in order to see what the best way to account for the differences in exposure, and the auto adjustments, but I believe the to workaround my inability to disable the auto exposure is to have the exposure adjustment weighted to affect the brightness value. Another approach, perhaps in conjunction with keeping track of the camera exposure value, is to access the phone’s light sensor, and have that value adjust the brightness reading. Again, I need to do a fair amount of testing to determine which method (or combination of methods) would enable the most accurate data readings.
After deciding to move forward with the basic functionality and to worry about the auto exposure problem later, I needed to get a handle on accessing the pixel data from the camera preview. There is a lot of information out there about doing image processing on pictures taken with the Android (to create filters), but actually doing live processing on the camera preview was a bit harder to find examples for. One important thing I learned is that the Android camera has a different pixel format, namely NV21, which has to be converted into RGB values in order to do normal pixel processing. Ironically, the method to convert the Android pixels into RGB data came out of the Ketai project, which wrote many of the sensor libraries for Processing for Android (which I decided to forego in favor of actual Android development). Here is the way they go about changing the NV21 pixels into useable RGB data:
static public void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width,
int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000)
| ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
}
After working through a few fairly helpful examples but not getting too far, I somehow googled the right thing to bring me here, to a pdf that went through the basic Android “Hello World” tutorial, but then also made available a camera app which drew histograms of the Red, Green, and Blue pixel values for each frame of video. Ironically, I made a similar project earlier on in Comp Cameras. Anyway, this code was extremely helpful to work from, as it was already analyzing the mean red, green and blue values, as well as drawing them over the camera preview layer. (Remember to grab the Manifest code as well or else it won’t run).
After getting this working on the phone, and changing the code to capture average brightness values of the whole frame, not the average RGB values of each pixel in the frame, I added the GPS tracking functionality (the Location Tracking example from Mobile Media was what I ended up working from). Then it was just a matter of uploading the data I was capturing to a server. With some help from people who know PHP much better than I do, I was able to write a short script that writes the data it receives from the mobile app to a text file that lives on the ITP server. The app sends data every 64 frames, which seems to be working pretty well for now. Here’s a zipped folder of my code in its current state. Feel free to download and upload to your phone as well. Just be sure to Force Quit the Application in the Settings Menu if you want to restart. Hmm, that’s another thing I need to fix.
Next steps:
- Improve UI and functionality.
- Add Thread for sending data to server
- Implement the ExposureCompensation functionality and/or the Light Sensor Data
- Possibly keep track of the Accelerometer data to tell how people are holding the phone in order to have more ability to throw out “bad data.”
- Get out there and start collecting data!
- Make the phone available as an Android App in order to start crowd sourcing the data collection.
- Start visualizing the data!
Here’s video of a dry run of the Darkness Map App in action:
