magic window
Surabhan Temiyabutr
A kind of remote display that makes use of "body interaction" and basic aspects of our perception to behave and hence, to feel like a real window to a faraway place.
Classes
Final Project Seminar
Keywords
computer graphics, telepresence, remote display, camera input, computer vision, view morphing
Description
A live image of a faraway place shows on a screen. The display changes _instantaneously_ and correspondingly with the viewer's left-right motion to create a sense of parallax.
The effect is similar to looking out a real window and seeing motion parallax as one moves side to side. The appropriately changing display in response to the viewer's position is intended to create a sense of connection with the remote space at a very basic perceptual/physical level.
A major reason the changing display will be effective is because change will happen instantaneously: faster than any video camera on a motor over a network or not. This will be very satisfying at a perceptual and interactive level.
The implementation of this effect is a technical feat: two stationary cameras placed along the same horizontal axis along with "view morphing" or "image-based rendering" algorithms will be used to generate intermediary images for the viewer. Similar techniques were used for the famous "virtual camera" scenes in the "Matrix" movies.
The effect is similar to looking out a real window and seeing motion parallax as one moves side to side. The appropriately changing display in response to the viewer's position is intended to create a sense of connection with the remote space at a very basic perceptual/physical level.
A major reason the changing display will be effective is because change will happen instantaneously: faster than any video camera on a motor over a network or not. This will be very satisfying at a perceptual and interactive level.
The implementation of this effect is a technical feat: two stationary cameras placed along the same horizontal axis along with "view morphing" or "image-based rendering" algorithms will be used to generate intermediary images for the viewer. Similar techniques were used for the famous "virtual camera" scenes in the "Matrix" movies.
User Scenario
The interaction is very simple, though the effect is intended to be an enjoyable and meaningful one. Simply look at the screen, perhaps moving back and forth for a different view of the scene presented in the display.
For the Show, the user may be sitting down in a table. This is necessary in order to get a wider camera view than usual. Sitting down will hopefully also keep the face-tracking manageable.
For the Show, the user may be sitting down in a table. This is necessary in order to get a wider camera view than usual. Sitting down will hopefully also keep the face-tracking manageable.
Implementation
The system consists of a PC with custom software, regular display, and webcam. The "full-version" uses a live networked dual video stream from a pair of rigged firewire cameras, but for demonstration purposes a prerecorded video will be used.
The system is built to run on my own custom-built PC: a dual-core (two processor) PC with a modern GPU (graphics card). The workload is divided three ways: two programs and a GPU program. This is necessary for efficiency and for interactive graphics: otherwise 3 networked PC's would be necessary for similar performance.
There is a program for headtracking and a program for video preprocessing and stereo matching. A GPU program takes this data as inputs and creates an image for the viewer. The OpenCV library is used for computer vision along with the Cg language for pixel shaders. The main implementation language is C and the system API is Win32 and OpenGL.
Other than a PC, it uses a regular display with a webcam for headtracking. Due to positioning of the webcam to capture a fairly wide view, the display may need to be placed on a desk.
Currently, the system works, but only on still images or 3D OpenGL scenes. Work is still being done to process video or a stream of images: this is nontrivial, as special preprocessing needs to be done.
The system is built to run on my own custom-built PC: a dual-core (two processor) PC with a modern GPU (graphics card). The workload is divided three ways: two programs and a GPU program. This is necessary for efficiency and for interactive graphics: otherwise 3 networked PC's would be necessary for similar performance.
There is a program for headtracking and a program for video preprocessing and stereo matching. A GPU program takes this data as inputs and creates an image for the viewer. The OpenCV library is used for computer vision along with the Cg language for pixel shaders. The main implementation language is C and the system API is Win32 and OpenGL.
Other than a PC, it uses a regular display with a webcam for headtracking. Due to positioning of the webcam to capture a fairly wide view, the display may need to be placed on a desk.
Currently, the system works, but only on still images or 3D OpenGL scenes. Work is still being done to process video or a stream of images: this is nontrivial, as special preprocessing needs to be done.
Additional Documents
- cam setup 2 - Main Image