Gavin Hackeling
Zena Koo
Contextual Interface for Remote Gestural Input
This application tracks the position of physical objects in a live camera stream on the user's mobile device. Users can then manipulate the tracked objects using gestures on their device's touchscreen.
http://gavinmhackeling.com/blog/2011/12/contextual-interface-for-remote-gestural-input-preview/
Classes
Introduction to Computational Media,Introduction to Physical Computing
Introduction to Computational Media,Introduction to Physical Computing
This Android application tracks the position of physical objects in the live video stream from the device's camera. The physical objects must have some identifiable visual feature to use as a fiduciary marker (these features could be a photograph, pattern, or a QR code-style marker). The application outputs a serial data stream that can be read and interpreted by a variety of applications.
If the tracked object has a Bluetooth radio, the application can open communications to the object, allowing the user to manipulate the object through multi-touch gestures on the Android device's touchscreen. The user can see her changes to the object in real-time and in visual context on the screen.
This application is useful when a user can see an object but cannot physically access it. The object could be an interactive billboard or a window display behind glass. Currently, we have built three demonstration applications. The first allows users to draw on a remote display by dragging on their touchscreen (http://vimeo.com/33092926 ). The second allows users to turn on/off objects that are plugged into an Arduino-controlled power strip by tapping on the object in the Android's camera stream. The third allows a user to unlock a box by drawing a pattern on the box\'s lid through the touchscreen.
This application is also suitable for instances in which a graphic GUI is not appropriate; for instance, in an arcade shooting gallery, this application could allow light guns to be replaced with the users' Android devices. Another application could allow users to tag annotations to specific parts of an image (eg, a user could view notes tagged to specific parts of a mural).
Background
The application uses Qualcomm's AR SDK for natural image tracking.
The Android application accesses the compiled tacking libraries through Java Native Interface.
The Android device communicates with remote objects using the Bluetooth SPP service.
Audience
This technology is intended for applications in which a static GUI is unsuitable. Some interactions may be improved or facilitated by allowing the user to see the object she is controlling in the object's physical context. For instance, if the user is instructing an object to move to a new location, tapping on a location in a live video stream may be more intuitive than using a static GUI.
User Scenario
The app opens a live stream from the device's camera upon launching. The user can then perform touch gestures on the objects tracked in the camera stream to manipulate them. Changes to the objects are reflected immediately in the camera stream.
We have provided several example applications:
1) Users may draw on a distant display screen (for example, a digital billboard or window display) by drawing on the image as viewed through their mobile device\\\\\\\\\\\\\\\'s camera stream. Users could also navigate through a gallery of images on a distant display by using \\\\\\\\\\\\\\\"swipe\\\\\\\\\\\\\\\" gestures while viewing the screen through their mobile device.
2) An \\\\\\\\\\\\\\\"unlock screen\\\\\\\\\\\\\\\" like those used on smartphones can be applied to a physical lock-box; a user may view the box through their mobile device and draw a pattern on the box\\\\\\\\\\\\\\\'s top to unlock it.
3) Users may toggle distant physical objects (for instance, a lamp or stereo) on and off by viewing the device through their mobile device and tapping on it.
Implementation
Qualcomm's tracking libraries are compiled as a native shared object that the Android application accesses through Java Native Interface.
The companion programs are built variously with Arduino, Processing, and C#.
Conclusion
We learned to think about interactions that could be improved by using a particular technology rather than interactions that a technology could be applied to.
If the tracked object has a Bluetooth radio, the application can open communications to the object, allowing the user to manipulate the object through multi-touch gestures on the Android device's touchscreen. The user can see her changes to the object in real-time and in visual context on the screen.
This application is useful when a user can see an object but cannot physically access it. The object could be an interactive billboard or a window display behind glass. Currently, we have built three demonstration applications. The first allows users to draw on a remote display by dragging on their touchscreen (http://vimeo.com/33092926 ). The second allows users to turn on/off objects that are plugged into an Arduino-controlled power strip by tapping on the object in the Android's camera stream. The third allows a user to unlock a box by drawing a pattern on the box\'s lid through the touchscreen.
This application is also suitable for instances in which a graphic GUI is not appropriate; for instance, in an arcade shooting gallery, this application could allow light guns to be replaced with the users' Android devices. Another application could allow users to tag annotations to specific parts of an image (eg, a user could view notes tagged to specific parts of a mural).
Background
The application uses Qualcomm's AR SDK for natural image tracking.
The Android application accesses the compiled tacking libraries through Java Native Interface.
The Android device communicates with remote objects using the Bluetooth SPP service.
Audience
This technology is intended for applications in which a static GUI is unsuitable. Some interactions may be improved or facilitated by allowing the user to see the object she is controlling in the object's physical context. For instance, if the user is instructing an object to move to a new location, tapping on a location in a live video stream may be more intuitive than using a static GUI.
User Scenario
The app opens a live stream from the device's camera upon launching. The user can then perform touch gestures on the objects tracked in the camera stream to manipulate them. Changes to the objects are reflected immediately in the camera stream.
We have provided several example applications:
1) Users may draw on a distant display screen (for example, a digital billboard or window display) by drawing on the image as viewed through their mobile device\\\\\\\\\\\\\\\'s camera stream. Users could also navigate through a gallery of images on a distant display by using \\\\\\\\\\\\\\\"swipe\\\\\\\\\\\\\\\" gestures while viewing the screen through their mobile device.
2) An \\\\\\\\\\\\\\\"unlock screen\\\\\\\\\\\\\\\" like those used on smartphones can be applied to a physical lock-box; a user may view the box through their mobile device and draw a pattern on the box\\\\\\\\\\\\\\\'s top to unlock it.
3) Users may toggle distant physical objects (for instance, a lamp or stereo) on and off by viewing the device through their mobile device and tapping on it.
Implementation
Qualcomm's tracking libraries are compiled as a native shared object that the Android application accesses through Java Native Interface.
The companion programs are built variously with Arduino, Processing, and C#.
Conclusion
We learned to think about interactions that could be improved by using a particular technology rather than interactions that a technology could be applied to.