Kinect

Enter the Kinect

Most cameras can tell what color you are.   To give expression to more of your body it might be more important to know where your body is rather than what color it is.  The kinect is a depth sensing camera can tell you not only the color of all the pixels  but also how far away it is from the camera.  The software that that comes with the Kinect (PC only) can go further and based on this depth information cull out elements of the scene, for instance 23 of your body parts .  You should be able to make some fun software that reacts to the position of all of your body parts.

enter-kinect

Using a Kinectron Server

The Kinect comes with a USB cable and you could conceivably just plug that into  your computer.   (If you are just after the raw depth of all the pixels, you can plug a kinect 1 into your mac and use processing which is very similar to p5)  If you are after the “anatomy” of people in front of the camera and other functionality you can’t get plug a kinect (made by Microsoft) directly into the Apple Computers .   Instead we are going to connect the Kinect to a PC running Kinectron software that will act as a server.  Although the set up seems more complicated with another machine and an internet connection,  Kinectron will make your code a lot simpler.

xbox-one-kinect +    laptop  +   waves

We will then just ask for any kind of Kinect info over a network from P5.js running on any brand of laptop.

mac

In class these PC parts will be all set up for you.  At most you might have to double click on the “Kinectron” icon on the desktop.  But in the future if you want to do this yourself at home, you need to plug a Kinect2 camera into a PC, download, install and run the kinectron software.  The instruction for how to do that are here.

Connecting to the Kinect in your p5 code.

Start by going to this sketch and use the duplicate menu to copy it to your account (If you are not using the web editor or want to start from scratch you will need to add a library file to get this to work*). You need to know the address of the “kinect server” running on the PC in order connect you code to it (unless you are running your p5 code on the PC that is also running the kinectron server software and connected to the kinect ).  The internet uses IP addresses, for instance “172.44.51.30” or “kinectron.itp.tsoa.nyu.edu” which are like street address of the machine.  But your have to further specify the  port number for instance “9001”which would be like the apartment number of the particular software on that machine you want to talk to.  Look at the Kinectron app on the PC and it will give you both the IP Address and the Port Number.  You need to put the address and the port into it into your p5 code in the line where the address variable is set.  Do this in the setup function.

Creating a Kinect Object.

Now you want to make a kinectron “object”, put it in a variable, and use it to connect to the kinectron server.  Making an object is something you might have done already with sounds or images.

Callbacks

The future is unpredictable particularly in the world of network connections and user interface both of which will figure into your project.  In programming there is a very civilized way to deal with an unpredictable future events using “callbacks.”  Basically you set up function name that that the computer will call for you anytime a particular future event happens.  For instance you may already have seen that p5 always makes a callback for you so that anytime the mouse is pressed a function called mousePressed or mouseDragged is called.

For the Kinect instead of mouseDragged we need to set up a callback function to happen every time some info comes back from the kinect.  In the setup function above we named the call back function “bodyTracked” and now we are obliged to create a function by that name.

Notice that when the kinectron calls this function for you, it also sends the payload of information and we are putting it into a variable we called body.  Inside of that body object there are many joints.

kinectron.SPINEBASE
kinectron.SPINEMID
kinectron.NECK
kinectron.HEAD
kinectron.SHOULDERLEFT
kinectron.ELBOWLEFT
kinectron.WRISTLEFT
kinectron.HANDLEFT
kinectron.SHOULDERRIGHT
kinectron.ELBOWRIGHT
kinectron.WRISTRIGHT
kinectron.HANDRIGHT
kinectron.HIPLEFT
kinectron.KNEELEFT
kinectron.ANKLELEFT
kinectron.FOOTLEFT
kinectron.HIPRIGHT
kinectron.KNEERIGHT
kinectron.ANKLERIGHT
kinectron.FOOTRIGHT
kinectron.SPINESHOULDER
kinectron.HANDTIPLEFT
kinectron.THUMBLEFT
kinectron.HANDTIPRIGHT
kinectron.THUMBRIGHT

Inside every joint are several ways to measure the X, Y and Z location and orientations of the joint, depthX and depthY might be the easiest to use.

You can drill down  into that variable from body to which joint and then which measurement using “dot notation.”

Most of  the measurements that you drill down to are in a range from 0 to 1 and you have to convert them to a range you like using the map function (or because it is between 0-1 just multiplying by the biggest value in the range will also work).

Here is the code all together.  You can see it action here.  Press duplicate to start your own version under your login.  Change the IP address and port number to match your kinect server and this should work.

Start by duplicating these examples:

I recently started having a problem where I had to type in http:// (not https://) in the url for the editor.  eg http://alpha.editor.p5js.org/projects/S1tYy5fxl

 

Adding the Kinectron Library

*If you are starting from scratch you will need to add a library file to get this to work.  You can do that by adding this line to the html file of your sketch.  You need to 1) click on the “>” symbol on the left, 2) click on the html file,  3)add the line and save.library