Mobile Me(dia)

Shawn Van Every Shawn.Van.Every@nyu.edu
Spring 2008
H79.2690.1

Week 7 - Mobile Application Development with Mobile Processing

There are quite a few options for mobile application development. Some of them geared towards specific manufacturors and devices (C++ for Symbian, Python for Symbian Series 60, Java for Android, Objective C for iPhone, .Net for WIndows Mobile) and some geared towards hitting a wide range of devices (Flash Lite, Java ME) but more limiting.

This week we will concentrate on developing with Java ME using Mobile Processing. Mobile Processing gives us a nice environment that is cross platform for developing mobile applications using Java. Using Java gives us the ability to run on a wide range of devices but as with the rest of this course we are going to focus on devices with media capture and internet access capabilities. Using Mobile Processing gives us a nice set of libraries and a familiar development environment.

Getting Started with Mobile Processing

Prerequisites

Mobile Processing as you might have guessed is an off shoot of Processing which is an open source IDE (integrated development environment) and set of libraries for programming in Java.

The mobile version (Mobile Processing) is the same thing, except instead of programming with straight Java we are developing Java Microedition (Java ME), the flavor of Java that was developed for mobile phones.

In order to get started with Mobile Processing we need three things, first is Java itself. Mac users should already have Java and therefore shouldn't need to download it. Windows users may need to. Furthermore, I would recommend the JDK (Java Development Kit) rather than just the runtime (which the above link will allow you to download).

The next thing that is required is a WTK (Wireless Toolkit). The WTK provides tools for building and emulating mobile apps. Sun makes a WTK for Windows machines.

Unfortunately, Sun does not have the WTK for the Mac but an independent developer has created one called the mpowerplayer SDK.

Last is Mobile Processing itself.

Configuration

After installing Java, the WTK and Mobile Processing the next step is to open up Mobile Processing and go to Preferences. In Preferences under "Mobile" you need to specify the location of your WTK install as well as choose a CLDC (Connected Limited Device Configuration) and MIDP (Mobile Information Device Profile) version. I recommend CLDC 1.1 and MIDP version 2.0. This configuration and profile specify the capabilities of the devices that you will be running the application you develop on.

More Information

Connected Limited Device Configuration - Wikipedia
Mobile Information Device Profile - Wikipedia
Mobile Processing Phones

Beginning Mobile Processing Development

A Simple App

Writing an application with Mobile Processing is very straight forward. Simply open up the application and simply start using the methods available.

Hello World:

import processing.core.*;		
		
public class HelloWorld extends PMIDlet
{
  
  PFont font;
  String hi;
  
  void setup()
  {
    font = loadFont("HelveticaNeue-24.mvlw");
    textFont(font);
    hi = new String("Hello World");
  }

  void draw()
  {
    text(hi,10,25);
  } 
}

You might notice that I added a bit more code than is nessecary for working with Mobile Processing (like Processing itself). The above could have simple been written as follows:

  PFont font;
  
  void setup()
  {
    font = loadFont("HelveticaNeue-24.mvlw");
    textFont(font);
  }

  void draw()
  {
    text("Hello World",10,25);
  }

The reason I wrote the top one such as I have is that it could be used in any IDE such as Eclipse that supports J2ME with the inclusion of the Mobile Processing libraries.

You will notice that in the top listing, we are extending PMIDlet which is a Mobile Processing class that extends the standard Java ME MIDlet class (the base of all Java Mobile Applications).

Testing and Installing

To test this application, you can simply hit the "Run" button in Mobile Processing which will compile the application and run it in your emulator (MPowerPlayer or Wireless Toolkit).

If it runs successfully there, you can choose "Export MIDlet" and transfer the application to your phone for testing. In order to test it, I typically send it to my phone through bluetooth on my Mac. On a PC you can do the same if you have bluetooth, otherwise you will have to connect via a cable and transfer it that way or upload it to the internet and download it through your phone's browser.

More Information

How to write apps with Mobile Processing on your N95 by our very own Josh Knowles
Mobile Processing >> Discourse - Building from Netbeans/Eclipse etc
Mobile Processing >> Examples

Additional Libraries

A variety of libraries exist for Mobile Processing. Everything from video and photo capture, mms messaging, sound and bluetooth are supported through the use of libraries.

Mobile Processing Libraries

To install libraries for use with Mobile Processing, you typically have to quit the application and move the folder enclosing the library folder to Mobile Processing's "libraries" folder. When you start the application back up, you should see the library listed under the Sketch, Import Library menu item.

I put together a couple of libraries that build upon things that we have been doing through the mobile web and MMS to email. They can be found here: mobvcasting >> Mobile Processing Libraries. Below are some examples that uses these libraries:

Media Capture and Playback

MVideoCapture Library
MVideoPlayback Library

Simple Video Capture and Playback:

import processing.core.*;
import com.mobvcasting.mvideocapture.*;
import com.mobvcasting.mvideoplayback.*;

public class MVideoCaptureTest extends PMIDlet
{
  MVideoPlayback vidplay;
  MVideoCapture vidcap;
  byte[] videoData;

  String captureKey = "Capture";
  String stopCaptureKey = "Stop Capture";
  String playKey = "Play Video";

  public void setup()
  {
    softkey(captureKey);

    vidcap = new MVideoCapture((PMIDlet)this);
    vidcap.showCamera();

    noLoop();
  }

  public void draw()
  {
  }

  public void softkeyPressed(String label)
  {
    if (label.equals(captureKey))
    {
      //vidcap.timedCapture(20);
      vidcap.startCapture();
      softkey(stopCaptureKey);
    }
    else if (label.equals(stopCaptureKey))
    {
      vidcap.stopCapture();
    }
    else if (label.equals(playKey))
    {
      vidplay = new MVideoPlayback(this,videoData,"video/3gpp");
      vidplay.showPlayer(0, 0, width, height);
      vidplay.playVideo();
    }
    redraw();
  }

  public void libraryEvent(Object library, int event, Object data)
  {
    if (library.getClass().isInstance(vidcap) && event == MVideoCapture.CAPTURE_COMPLETE)
    {
      videoData = (byte[])data;
      vidcap.hideCamera();
      vidcap.closeCamera();

      softkey(playKey);
      redraw();
    }
  }
}

Testing this app is a bit tricky since the emulators are unable to emulate video capture (and playback of captured video for that matter). In order to test it, you need to "Export MIDlet" and transfer it to your phone (typically through bluetooth).

One thing to note about this example is that you can change the format of the video that is recorded by changing the way you instantiate the MVideoCapture and MVideoPlayback libraries. In this example the MVideoCapture library is instantiated with the default options (nothing is specified) but you could instantiate it as follows to get higher quality video (if your phone supports it):

vidcap = new MVideoCapture((PMIDlet)this,"capture://video?encoding=video/mp4");

There are other possibilities as well for specifying other aspects of the capture (such as height and width). You can see those possibilities listed here: MMAPI Manager: Locators for Live Media Capture

Media Uploading

MHTTPFilePoster Library

Taking the above example a step further we can utilize the MHTTPFilePoster library in combination with our File Uploading PHP script from Week 3.

import processing.core.*;

import com.mobvcasting.mvideocapture.*;
import com.mobvcasting.mvideoplayback.*;
import com.mobvcasting.mhttpfileposter.*;

public class MVideoCaptureTest extends PMIDlet
{
  MHTTPFilePoster poster;
  MVideoPlayback vidplay;
  MVideoCapture vidcap;
  
  byte[] videoData;

  String captureKey = "Capture";
  String stopCaptureKey = "Stop Capture";
  String playKey = "Play Video";
  String stopPlayKey = "Stop Video";
  String uploadKey = "Upload Video";
  String uploadStatus = "Check Upload";
  
  String statusMessage = "Not Captured";
  
  PFont font;
 
  public void setup()
  {
    softkey(captureKey);

    font = loadFont("Arial-Black-12.mvlw");
    textFont(font);

    vidcap = new MVideoCapture((PMIDlet)this);
    vidcap.showCamera();

    noLoop();
  }

  public void draw()
  {
    background(255);
    text(statusMessage,10,15);    
  }

  public void softkeyPressed(String label)
  {
    if (label.equals(captureKey))
    {
      //vidcap.timedCapture(20);
      vidcap.startCapture();
      softkey(stopCaptureKey);
    }
    else if (label.equals(stopCaptureKey))
    {
      vidcap.stopCapture();
    }
    else if (label.equals(playKey))
    {
      vidplay = new MVideoPlayback(this,videoData,"video/3gpp");
      vidplay.showPlayer(0, 0, width, height);
      vidplay.playVideo();
      softkey(stopPlayKey);
    }
    else if (label.equals(stopPlayKey))
    {
      vidplay.stopVideo();
      softkey(uploadKey);
    }
    else if (label.equals(uploadKey))
    {
      // Setup the fields we need for the form
      /*
	Title: <input type="text" name="subject" /><br />
	Description: <input type="text" name="message_text" /><br />
	<input type="file" name="bytes" />
	<input type="hidden" name="form_submitted" value="true" />
	<br />
        <input type="submit" name="submit" value="submit" />
      */
      
      String[][] formvars = new String[4][2];
      formvars[0][0] = "subject";
      formvars[0][1] = "the subject";
      formvars[1][0] = "message_text";
      formvars[1][1] = "the message";
      formvars[2][0] = "form_submitted";
      formvars[2][1] = "true";
      formvars[3][0] = "submit";
      formvars[3][1] = "submit";
      
      poster = new MHTTPFilePoster(this, "http://your.server/path/upload.php", "avideofile", "bytes", videoData, formvars);
      poster.startUpload();
      statusMessage = "Uploading Video";
      softkey(uploadStatus);
    }
    else if (label.equals(uploadStatus))
    {
      statusMessage = poster.getStatus() + " " + poster.getServerResponse();
    }
    redraw();
  }

  public void libraryEvent(Object library, int event, Object data)
  {
    if (library.getClass().isInstance(vidcap) && event == MVideoCapture.CAPTURE_COMPLETE)
    {
      videoData = (byte[])data;
      vidcap.hideCamera();
      vidcap.closeCamera();

      softkey(playKey);
      redraw();
    }    
  }
}

If you get a "java.lang.RuntimeException 0" when trying to run this directly after installing it, chances are that you didn't include the font before exporting ("Arial-Black-12.mvlw").

A couple of notes regarding the code above. The first is this array:

      String[][] formvars = new String[4][2];
      formvars[0][0] = "subject";
      formvars[0][1] = "the subject";
      formvars[1][0] = "message_text";
      formvars[1][1] = "the message";
      formvars[2][0] = "form_submitted";
      formvars[2][1] = "true";
      formvars[3][0] = "submit";
      formvars[3][1] = "submit";

It is a two dimensional array of form variables that need to be posted along with the file upload to make the form that the file is going to function correctly.

In this case, I am implementing all of the required elements for our PHP Upload form from week 2:

	Title: <input type="text" name="subject" /><br />
	Description: <input type="text" name="message_text" /><br />
	<input type="file" name="bytes" />
	<input type="hidden" name="form_submitted" value="true" />
	<br />
        <input type="submit" name="submit" value="submit" />

In the instantiation of the MHTTPPoster object I am specifying "bytes" as the name of the name of the form element for the file that is to be uploaded and specifying "avideofile" as the name of the file (it should probably include the extension so that it gets saved right on the server side.

 poster = new MHTTPFilePoster(this, "http://your.server/path/upload.php", "avideofile", "bytes", videoData, formvars);

Here is that element from the upload form itself:

				<input type="file" name="bytes" />

Last, you will notice that code is using the softkey function therefore utilizing soft keys to trigger actions. In Mobile Processing, when soft keys are pressed it triggers a callback to the softkeyPressed() method. This method is driving the mode application by checking which key was pressed and updating what is done based upon that.

MMS Sending

MMMSMessaging Library

Alternatively, we could use the MMMSMessaging library and send the video through that and use our PHP Mail Popper (parseMailScript) from Week 2 to receive the video.

import processing.core.*;

import com.mobvcasting.mvideocapture.*;
import com.mobvcasting.mvideoplayback.*;
import com.mobvcasting.mmmsmessaging.*;

public class MVideoCaptureTest2 extends PMIDlet
{
  MMMSMessaging mmssender;
  MVideoPlayback vidplay;
  MVideoCapture vidcap;
  
  byte[] videoData;

  String captureKey = "Capture";
  String stopCaptureKey = "Stop Capture";
  String playKey = "Play Video";
  String stopPlayKey = "Stop Video";
  String sendKey = "Send Video";
  
  String statusMessage = "Not Captured";
  
  PFont font;
 
  public static int MAX_MESSAGE_SIZE = 305000;

  public void setup()
  {
    softkey(captureKey);

    font = loadFont("Arial-Black-12.mvlw");
    textFont(font);

    vidcap = new MVideoCapture((PMIDlet)this);
    vidcap.showCamera();

    mmssender = new MMMSMessaging(this);

    noLoop();
  }

  public void draw()
  {
    background(255);
    text(statusMessage,10,15);    
  }

  public void softkeyPressed(String label)
  {
    if (label.equals(captureKey))
    {
      //vidcap.timedCapture(20);
      vidcap.startCapture();
      softkey(stopCaptureKey);
    }
    else if (label.equals(stopCaptureKey))
    {
      vidcap.stopCapture();
    }
    else if (label.equals(playKey))
    {
      vidplay = new MVideoPlayback(this,videoData,"video/3gpp");
      vidplay.showPlayer(0, 0, width, height);
      vidplay.playVideo();
      softkey(stopPlayKey);
    }
    else if (label.equals(stopPlayKey))
    {
      vidplay.stopVideo();
      softkey(sendKey);
    }
    else if (label.equals(sendKey))
    {
	if (videoData.length < MAX_MESSAGE_SIZE)
	{
	  mmssender.sendMMS("youremailaddressorphonenumber","subject","body",videoData,"video/3gpp","avideo.3gp");
	}
	else
	{				
	  // Too big, divide up
	  int numSegments = (int)Math.ceil((double)((double)videoData.length/(double)MAX_MESSAGE_SIZE));
	  for (int i = 0; i < numSegments; i++)
	  {
	    byte[] newOutputArray;
					
	    if (i < numSegments - 1)
	    {
		newOutputArray = new byte[MAX_MESSAGE_SIZE];
		System.arraycopy(videoData, i*MAX_MESSAGE_SIZE, newOutputArray, 0, MAX_MESSAGE_SIZE);
	    }
	    else
	    {
		newOutputArray = new byte[videoData.length - i*MAX_MESSAGE_SIZE];
		System.arraycopy(videoData, i*MAX_MESSAGE_SIZE, newOutputArray, 0, newOutputArray.length);
	    }
	    mmssender.sendMMS("youremailaddressorphonenumber","subject part " + (i+1) + " of " + numSegments,"body",newOutputArray,"video/3gpp","avideo_part_" + i + ".3gp");
          }
	}
        statusMessage = "Sending Video";
    }
    redraw();
  }

  public void libraryEvent(Object library, int event, Object data)
  {
    if (library.getClass().isInstance(vidcap) && event == MVideoCapture.CAPTURE_COMPLETE)
    {
      videoData = (byte[])data;
      vidcap.hideCamera();
      vidcap.closeCamera();

      softkey(playKey);
      redraw();
    }    
  }
}

The key to using the MMMSMessaging library is the sendMMS method as shown below. The first argument is the email address or phone number that you would like to send the message to. The second argument is the subject of the message, the third the text portion of the message, the fourth is the byte array for the file, the fifth the "mime-type" of the file and the six the name of the file.

mmssender.sendMMS("emailaddressorphonenumber","subject","body",videoData,"video/3gpp","avideo.3gp");

One extra thing that I threw into this example is the ability to divide up the file to be sent if it is too big to be sent through MMS in one go. You should notice the constant which is called MAX_MESSAGE_SIZE. In this code it is set to 305000 which is just about 300KB the standard limit for MMS (though it is different on different phones).

This is the section of code which does the breaking up of the message. It requires you to take some extra steps in the parseMailScript which we aren't going to cover just yet (I will send out an email with the additions to make it work right).

	if (videoData.length < MAX_MESSAGE_SIZE)
	{
	  mmssender.sendMMS("youremailaddressorphonenumber","subject","body",videoData,"video/3gpp","avideo.3gp");
	}
	else
	{				
	  // Too big, divide up
	  int numSegments = (int)Math.ceil((double)((double)videoData.length/(double)MAX_MESSAGE_SIZE));
	  for (int i = 0; i < numSegments; i++)
	  {
	    byte[] newOutputArray;
					
	    if (i < numSegments - 1)
	    {
			newOutputArray = new byte[MAX_MESSAGE_SIZE];
			System.arraycopy(videoData, i*MAX_MESSAGE_SIZE, newOutputArray, 0, MAX_MESSAGE_SIZE);
	    }
	    else
	    {
			newOutputArray = new byte[videoData.length - i*MAX_MESSAGE_SIZE];
			System.arraycopy(videoData, i*MAX_MESSAGE_SIZE, newOutputArray, 0, newOutputArray.length);
	    }
	    mmssender.sendMMS("youremailaddressorphonenumber","subject part " + (i+1) + " of " + numSegments,"body",newOutputArray,"video/3gpp","avideo_part_" + i + ".3gp");
      }
	}

If you don't want the message divided up in this manner, you should remove the code under the "else" and instead give a message to the user stating that the file is too big.