SketchChat – Keru Wang

Description

Videoconferencing has gained popularity because it is not always feasible for collaborators to meet in person. It is well suited for collaborative 2D tasks such as shared document editing, but lacks the spatial awareness and immersion required for collaborative 3D tasks. To solve this problem, researchers recreate the shared immersive world for remote meetings via dedicated hardware like AR/VR headsets or using compute-intensive technologies like 3D human reconstruction. This makes the system not easy to access. We propose SketchChat, an immersive online conferencing platform that seamlessly incorporates 2D and 3D multimodal collaborations into a video chat metaphor that requires only a laptop computer. SketchChat renders the collaborators as 2D cropped video feeds collocated in a 3D meeting space, while reconstructing their hands in 3D for intuitive spatial interaction. Features including animated sketching, custom coding and widgets, and object manipulation are supported by multimodal input for smooth multidimensional collaboration.

Video