Computational Cameras Final Project – Darkness Map Mobile App

For my Computational Cameras final, I set out to build an Android application that would serve as the sensor device for the Darkness Map project. To my delight, I was actually able to get something working via Android. As of now, I have an app running on my phone which analyzes the average brightness of each frame of the camera preview. It also keeps track of the time stamp, longitude, latitude, and accuracy of the GPS. These values get stored up in an array and sent to a server every 64 frames (so as not to overload the server or slow the app down by constantly sending values over the network). You can find the data online at itp.nyu.edu/~gh726/darknessmap/rawdata/longdata.txt.

One of the hurdles that I encountered is that most Android phones don’t have nice enough cameras that can disable the auto-exposure function. In order for the Darkness Map to have data that’s somewhat accurate, I need to disable the auto-adjustments so that the camera doesn’t adjust the exposure down when it sees a very bright area, or adjust up when it’s in darkness. After searching around the internet for a long time, I realized that this was going to be hard to control via hardware, mainly because most camera phones are built so that they have the opposite capability, to deliver the best picture quality in a variety of lighting conditions. Shawn Van Every, who teaches Mobile Media, was kind enough to point me to the getExposureCompensation() function, which returns the value that the camera exposure is adjusting for. If this function returns 0, then camera exposure adjustment is not supported (which is ideal for my purposes). However, my phone’s camera adjusts between a range of -4 to 4.

So, I haven’t implemented this functionality yet, but I believe that keeping track of the amount that the phone is adjusting the exposure will allow me to weight the brightness readings by taking into account the degree of exposure adjustment the camera made. For example, if the camera returns a midrange brightness reading of 100, but the Exposure Compensation was a full 4, then I’m pretty sure it’s actually a lot darker than a value of 100 would indicate. I need to do a lot more testing and play around with various equations in order to see what the best way to account for the differences in exposure, and the auto adjustments, but I believe the to workaround my inability to disable the auto exposure is to have the exposure adjustment weighted to affect the brightness value. Another approach, perhaps in conjunction with keeping track of the camera exposure value, is to access the phone’s light sensor, and have that value adjust the brightness reading. Again, I need to do a fair amount of testing to determine which method (or combination of methods) would enable the most accurate data readings.

After deciding to move forward with the basic functionality and to worry about the auto exposure problem later, I needed to get a handle on accessing the pixel data from the camera preview. There is a lot of information out there about doing image processing on pictures taken with the Android (to create filters), but actually doing live processing on the camera preview was a bit harder to find examples for. One important thing I learned is that the Android camera has a different pixel format, namely NV21, which has to be converted into RGB values in order to do normal pixel processing. Ironically, the method to convert the Android pixels into RGB data came out of the Ketai project, which wrote many of the sensor libraries for Processing for Android (which I decided to forego in favor of actual Android development). Here is the way they go about changing the NV21 pixels into useable RGB data:

static public void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width,
			int height) {
		final int frameSize = width * height;

		for (int j = 0, yp = 0; j < height; j++) {
			int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
			for (int i = 0; i < width; i++, yp++) {
				int y = (0xff & ((int) yuv420sp[yp])) - 16;
				if (y < 0)
					y = 0;
				if ((i & 1) == 0) {
					v = (0xff & yuv420sp[uvp++]) - 128;
					u = (0xff & yuv420sp[uvp++]) - 128;
				}

				int y1192 = 1192 * y;
				int r = (y1192 + 1634 * v);
				int g = (y1192 - 833 * v - 400 * u);
				int b = (y1192 + 2066 * u);

				if (r < 0)
					r = 0;
				else if (r > 262143)
					r = 262143;
				if (g < 0)
					g = 0;
				else if (g > 262143)
					g = 262143;
				if (b < 0)
					b = 0;
				else if (b > 262143)
					b = 262143;

				rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000)
						| ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
			}
		}
	}

After working through a few fairly helpful examples but not getting too far, I somehow googled the right thing to bring me here, to a pdf that went through the basic Android “Hello World” tutorial, but then also made available a camera app which drew histograms of the Red, Green, and Blue pixel values for each frame of video. Ironically, I made a similar project earlier on in Comp Cameras. Anyway, this code was extremely helpful to work from, as it was already analyzing the mean red, green and blue values, as well as drawing them over the camera preview layer. (Remember to grab the Manifest code as well or else it won’t run).

After getting this working on the phone, and changing the code to capture average brightness values of the whole frame, not the average RGB values of each pixel in the frame, I added the GPS tracking functionality (the Location Tracking example from Mobile Media was what I ended up working from). Then it was just a matter of uploading the data I was capturing to a server. With some help from people who know PHP much better than I do, I was able to write a short script that writes the data it receives from the mobile app to a text file that lives on the ITP server. The app sends data every 64 frames, which seems to be working pretty well for now. Here’s a zipped folder of my code in its current state. Feel free to download and upload to your phone as well. Just be sure to Force Quit the Application in the Settings Menu if you want to restart. Hmm, that’s another thing I need to fix.

Next steps:

- Improve UI and functionality.
- Add Thread for sending data to server
- Implement the ExposureCompensation functionality and/or the Light Sensor Data
- Possibly keep track of the Accelerometer data to tell how people are holding the phone in order to have more ability to throw out “bad data.”
- Get out there and start collecting data!
- Make the phone available as an Android App in order to start crowd sourcing the data collection.
- Start visualizing the data!

Here’s video of a dry run of the Darkness Map App in action: