Redial: Interactive Telephony : Week 9

Getting outside the phone: Displays

We have talked quite a bit about the power of the voice and our relationship to it as humans. Quite a bit is being done at ITP in the realm of new interfaces and displays which is equally as exciting. In this class we will explore the possibilities for combining these.

Displays

Processing is great for developing single user, rich graphical applications. What if we could use processing to drive a multiple user display that was controllable by a simple phone call?

Asterisk + Processing

Dan Shiffman put together a nice tutorial and a bunch of sample code that helps us get started.

In his example, Dan is using Java as the language to create an AGI script (as opposed to PHP which we have been using up to this point). He is also using something called EAGI which is the Enhanced Asterisk Gateway Interface which allows us to get at the actual audio data of the call (normal AGI does not).

Unfortunately, there is a little quirk with using Java and EAGI in that Java needs to open a special file on the file system that allows it to access the live audio stream. This file is identified by the "process id" of the EAGI application and Java does give us the means to access this information. Therefore we have to use a little shell script to launch the Java process instead of launching the Java application directly.

NET-ID_extensions.conf:
[redial_NET-ID]
exten => s,n,EAGI(/path/toyour/shellscript/runEAGI.sh);	Pointing to shell script with EAGI
		
Shell Script (runEAGI.sh):
#!/bin/sh
/usr/java/jdk1.5.0_10/bin/java -classpath /path/toyour/javaclass/ JEAGIClient $$		
		
The $$ in the shell script is the actual process id and it is being passed to the Java application as if it were a command line argument.

To actually run this example we first need to download Dan's example files: JEAGI Client: JEAGIClient.java, Client.java
JEAGI Server: JEAGIServer.java, ServerThread.java Processing examples: Asterisk.pde, AsteriskMulti.zip

Since EAGI/AGI processes are per call, we need to have a server running somewhere on the server deal with connections from Processing as well as from Asterisk and hand them back and forth. Dan created a JAEAGIServer.java application (which relies upon ServerThread.java) to communicate back and forth.

Servers need a port to listen on. In order for us not to step on eachother's toes, we need to use unique ports for servers. For this class we should use 9000 + our extension. For me this would be 9010, for you it might be 9232 or something like that.

You need to change this port in each of the applications (client, server and processing):

JEAGIServer:
int portNum = 9010;

Asterisk.pde:
client = new Client(this, "yourserver", 9010); // Port is whatever you set it up to be

JEAGIClient:
JEAGIClient jeagi = new JEAGIClient(Integer.parseInt(args[0]),"localhost",9010);


Finally we need to compile and run..

To compile Java on our Asterisk server we need to point to:
/usr/java/jdk1.5.0_10/bin/javac
This is the javac compiler (there is a built-in one that can just be used by calling javac but this one is newer).
Run this command in the directory containing all of the .java files from Dan:

/usr/java/jdk1.5.0_10/bin/javac *.java		
		
Now we can start up the server process. Dan put the server class into a "package" so in order to run it we need to create a foler called "server" and then run it:
[sve204@asterisk javaagi]$ mkdir server
[sve204@asterisk javaagi]$ mv JEAGIServer.class server/.
[sve204@asterisk javaagi]$ mv ServerThread.class server/.
[sve204@asterisk javaagi]$ /usr/java/jdk1.5.0_10/bin/java server.JEAGIServer
Server running at asterisk.itp.tsoa.nyu.edu/128.122.151.44  9010
Waiting for a new connection: 		
		
Assuming I setup my extensions.conf correctly, any calls that come in should show show up in the terminal window for this server. (You will probably also have to make the runEAGI.sh script executable by asterisk.)
/127.0.0.1:50968 connected.
Waiting for a new connection: 
reading: sve204_gizmo,newcall
broadcasting: sve204_gizmo,newcall
reading: sve204_gizmo,2
broadcasting: sve204_gizmo,2
reading: sve204_gizmo,6
broadcasting: sve204_gizmo,6
reading: sve204_gizmo,#
broadcasting: sve204_gizmo,#
reading: sve204_gizmo,8
broadcasting: sve204_gizmo,8		
		
Now we can run the processing code and connect to the same server.

First we need to configure the processing applet to connect to the correct server on the correct port:
  client = new Client(this, "asterisk.itp.tsoa.nyu.edu", 9010);  // Port is whatever you set it up to be and asterisk.itp.tsoa.nyu.edu is the host name		
		
In Dan's example, you will notice a snippet of code that looks like this:
if (client.available() > 0) { 
messageFromServer = client.readString();   // Read it as a String 
} 		
		
This is the snippet that receives the communication from the server. It comes in in the following format:
callerid,digitpressed	
		
We can do a bit more than he is doing in his example by splitting these up and doing different things based on the digit that is pressed. Here is an example which moves a ball around the screen:
import processing.net.*; 

// Declare a client 
Client client; 

// A String to hold whatever the server says 
String messageFromServer = "Waiting for call.";
PFont f; 

int x, y;

void setup() { 
  size(600,200); 
  frameRate(30); 
  // Create the Client 
  client = new Client(this, "asterisk.itp.tsoa.nyu.edu", 9010);  // Port is whatever you set it up to be and asterisk.itp.tsoa.nyu.edu
  background(0); 
  x = width/2;
  y = height/2;
} 

void draw() 
{ 
  background(0); 
  // Display message from server 
  fill(255); 
  ellipse(x,y,50,50);

  // If there is information available to read from the Server 
  if (client.available() > 0) { 
    messageFromServer = client.readString();   // Read it as a String 
    String[] parts = split(messageFromServer,",");
    if (parts.length > 0)
    {
      String part = trim(parts[1]);
      if (part.equals("1"))
      {
         x-=10; 
         y-=10;        
      }
      else if (part.equals("2"))
      {
         y-=10; 
      }
      else if (part.equals("3"))
      {
         y-=10; 
         x+=10;
      }
      else if (part.equals("*"))
      {
         x-=10; 
         y+=10;
      }
      else if (part.equals("0"))
      {
         y+=10; 
      }
      else if (part.equals("#"))
      {
         x+=10; 
         y+=10; 
      }
    }
  } 
} 
		
Dan also has an example which works with multiple callers.. Give it a shot..

Doing More

The Java application JEAGIClient opens up communication with Asterisk and can issue commands just like our PHP AGI applications can. Unfortunately, there doesn't exist an appropriate library (like PHPAGI) for making this easy so we have to fall back on using straight AGI commands (which are documented in the Asterisk TFOT book as well as on the voip-info.org wiki.

In Dan's example he is only using the "WAIT FOR DIGIT" command but obviously we could use any of the commands such as "STREAM FILE".

The documentation for STREAM FILE on the voip-info.org wiki shows the following:
STREAM FILE <filename> <escape digits>
		
Returns: 
failure: 200 result=-1 endpos=<sample offset> 
failure on open: 200 result=0 endpos=0 
success: 200 result=0 endpos=<offset>
digit pressed: 200 result=<digit> endpos=<offset>
which is slightly different from the WAIT FOR DIGIT command
Usage: WAIT FOR DIGIT <timeout>

Waits up to <timeout> milliseconds for channel to receive a DTMF digit. 

Use -1 for the <timeout> value if you desire the call to block indefinitely. 

Returns: 
failure: 200 result=-1 
timeout: 200 result=0 
success: 200 result=<digit> 

<digit> is the ascii code for the digit received. 
Most of the return code is the same so we should be able to substitute STREAM FILE for WAIT FOR DIGIT in his code. Better yet, we could create another method to do that instead and call that in our code. Here is an untested implementation of that:
public void streamFile(String filetostream) {
	try {
		// Stream a file..
		String wait = "STREAM FILE " + filetostream + " #*0123456789\n";
		out.write(wait.getBytes());

		String line = null;
		while ((line = bin.readLine()) != null) {

			String agi_callerid = "agi_callerid:";
			int index = line.indexOf(agi_callerid);
			// See if we can get the phone number (for a new call)
			if (index > -1) {
				id = line.substring(index + agi_callerid.length()+1,line.length());
				client.send(id + ",newcall");
			}
			// The message we want starts with 200, but does not = 1
			else if (line.indexOf("200") == 0 && !line.equals("200 result=1")) {
				// Parse out what digit was pressed
				String result = "200 result=";
				int i = line.indexOf(result);
				if (i > -1) {
					String key = line.substring(i+result.length(),line.length());
					// -1 means hangup
					if (key.equals("-1")) {
						client.send(id+",hangup");
					} else {
						// If this doesn't work we'll send "error"
						String c = "error";
						try {
							// Convert to integer
							c = "" + (char) Integer.parseInt(key);
						} catch (Exception e) {
							// oops
						}
						client.send(id+","+c);
						// tfsos.write(("Message: "+id+","+c+"\n").getBytes());
					}
				}
				out.write(wait.getBytes());
			}
		}
	} catch (Exception e) {
		// There was some sort of error
		System.out.println("SAY ALPHA ERROR \"*#\"");
		System.out.println(e);
		e.printStackTrace();
	}
	//client.quit();  // Take out the quit..
}
		


Finally we will have to change what is called in the main to be our new function first:
//jeagi.streamFile("/path/to/file");
jeagi.streamFile("/var/lib/asterisk/sounds/vm-extension");
jeagi.waitForDigits();
		
instead of just
jeagi.waitForDigits();
		

Asterisk + Flash

As with using Asterisk to interact with a Processing sketch, it can also be used to communicate with a Flash application running on the web or elsewhere.

In fact, the only thing we need to change is one line in the server code that Dan has provided us: (Line 48)
                       thisConnection.sendToThisClient(_text + "\0");// \0 for flash EOL
		
and now we can use the XMLSocket AS3 class in our flash application.
var xmls=new XMLSocket();
xmls.connect("asterisk.itp.tsoa.nyu.edu",9001);
xmls.addEventListener(Event.CONNECT,xmlsocketEvent);
xmls.addEventListener(Event.CLOSE,xmlsocketEvent);
xmls.addEventListener(IOErrorEvent.IO_ERROR,xmlsocketEvent);
xmls.addEventListener(DataEvent.DATA, xmlsocketEvent);

function xmlsocketEvent(event):void
{
	switch(event.type)
	{
		case 'ioError':
			trace("ioError");
			break;
		case 'connect':
			trace("Connected");
			break;
		case 'close':
			trace("Disconnected");
			break;
		case 'data':
			trace("data" + event);
	}
}
		
You can download a full flash example.

The above shows how to break out of asterisk and control processing or flash. The same methods could be used for pretty much any network communication that you want to occur in real time. In the next couple of weeks we will look at streaming audio out of asterisk (using a similar method) as well as allowing more than just one way interaction.

Examples

MegaPhone