PoseNet in Tensorflow.js: real-time human pose estimation in the browser.

In May of this year, in collaboration with Google Creative Lab, I released a TensorFlow.js version of PoseNet, a machine learning model which allows for real-time human pose estimation in the browser.

Try a live demo here

Pose estimation has many uses, from interactive installations that react to the body to augmented reality, animation, fitness uses, and more. While many alternate pose detection systems have been open-sourced, all require specialized hardware and/or cameras, as well as quite a bit of system setup. With PoseNet running on TensorFlow.js anyone with a decent webcam-equipped desktop or phone can experience this technology right from within a web browser.

Underneath the hood, PoseNet uses a convolutional neural network and algorithms to estimate single or multiple poses. The complexity of this has been extracted away into easy to an easy to use API; Amateur developers can tinker with and use this technology with just a few lines of code, without needing to understand anything about machine learning.

This session will go through

• Demos of PoseNet, and a high-level explanation of the algorithms, and the effects that changing the parameters have on speed and accuracy.
• An explanation of what Tensorflow.js is and why as a creative you would want to do machine learning in javascript in the browser.
• An overview of other way pose estimation has been done (Kinect, OpenPose), and how PoseNet does things differently.
• Workshop on how to get PoseNet up and running with a few lines of code in p5.

Audience

Up until the part about getting PoseNet up and running, this is for anyone that is interested in emerging technology and artificial intelligence. The workshop section will be geared towards those with basic understanding of Javascript and p5.