Telepresence Experiment

For my final project for Rest of You, I experimented with a new way to teleconference. Rather, I took the old way and added — and subtracted — a couple of key elements.

In an article written in InformationWeek in June, the president of IT consultancy Yeoman Technology Group, Mike, said that he only used video conferencing for casual discussions. He’s found that the prolonged, formal discussions are difficult because there’s a greater pressure on visual cues, like eye contact, in high-intensity situations.

In fact, even though video conferencing is readily available, a majority of workers simply do not want to use teleconferencing tools. It seems like a great solution for those who cannot attend a business meeting, but it’s hugely unpopular. Human interactions are quite complex, and there’s something about the current video-conferencing model that doesn’t seem to work.

John Tang is a researcher at Microsoft and works on the Embodied Social Proxy (ESP) project. ESP is just a computer on a cart, but it occupies the space that the person would have occupied in a team meeting. He’s hitting on something but I believe there are other reasons why teleconferencing does not work: 1) Those who are teleconferencing in are stuck inside a box — a computer screen, 2) we can clearly see that the person is not in the meeting room with us, but somewhere on a beach sipping a margarita and 3) because of the way the camera is situated, you can’t meet that person’s gaze. Simply put, I don’t think video chatting is real enough.

So for my final project, I attempted to tackle these three issues. I first used a kinect to subtract my background. Then, I hid in a closet while I projected my image onto a high-back chair in the classroom. I situated the projector so that my image was life-sized. The “chair” (which I made out of foam core) had a little embedded camera int the middle so my real self could see the classroom. I then used skype so that we could hear each other.

This worked pretty well, but it still did not solve the issue of the eye-gaze. I couldn’t turn to look at people in the room.  So, I rigged up the chair with a servo motor and set up a synchronous communication system where if I pushed a button on my keyboard, I could swivel the chair right and left from across the internet. For me, coming from a non-programming background, that proved to be the most challenging part (but it works and the code is below!). Please note that this code has been adapted from here.

In theory, it sounds swell. I do feel it could solve a lot of problems with teleconferencing. In practice, it was a nightmare to set up. I was using 4 computers — 2 for the meeting room to project my image, operate the chair and have a skype session, and 2 on my side to operate the kinect and to view the classroom. Plus, the projector had to be set up just right in order for my image to be life-sized.

I do think that this could be useful if everything is more consolidated into one machine — perhaps a super chair with everything, including the projector, embedded in it. I hope to explore this topic more next semester in my Spatial Media class.

CODE

**Kinect Code in Processing**

import ddf.minim.*;
import ddf.minim.signals.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;

import librarytests.*;
import org.openkinect.*;
import org.openkinect.processing.*;

Minim minim;


Kinect kinect;
int[] depth;

int frontThreshold = 500;
int backThreshold = 800;

int depthXPicOffset = 30;
int depthYPicOffset = 30;

int w = 640;
int h = 480;

void setup() {
  background(0);
  size(w, h);
  kinect = new Kinect(this);
  kinect.start();
  kinect.enableDepth(true);
  kinect.enableRGB(true);
  kinect.processDepthImage(false);
}


void draw() {
  background(0);
  loadPixels();
  fill(0);
  PImage myImage = kinect.getVideoImage();
  
  // Get the raw depth as array of integers
  int[] depth = kinect.getRawDepth();
  // We're just going to calculate and draw every 4th pixel (equivalent of 160x120)
  int skip = 1;

  for (int x = 0; x < w; x += skip) {
	for (int y = 0; y < h; y += skip) {
	  int offset = x + y * w;
	  int rawDepth = depth[offset];
		if (rawDepth > frontThreshold && rawDepth < backThreshold) {
		  int imageOffset = x - depthXPicOffset + (y + depthYPicOffset) * w;
		  if (imageOffset < myImage.pixels.length) (pixels[offset] )  = myImage.pixels[imageOffset] ;
				}
			}
		}
		updatePixels();

	}
  

**Server-side code in Processing to Control Chair rigged with Arduino and Servo Motor**

**Client-side code to receive commands from Server**