Kai Hung

World Loader is a machine learning powered Unity plugin which aims to accelerate 3D content production workflow. It takes a single image as an input and generates the environment reflect it through objection detection technique. The result will be a scene compose by independent models so 3D artist can edit the scene after generated.

https://github.com/kaichehung/WorldLoader

Description

World Loader is a machine learning powered Unity plugin which aims to accelerate the workflow of 3D content production, and it’s also a prototype of the futuristic content creation tool. The World Loader takes an image as input and output a 3D scene which reflects the environment in the picture. The objects in the generated scene are stand-alone models from the asset and can be edited and swapped, so it’s not just a single mesh with pixels as textures on it. It conserves the context of the environment. Therefore, animation artist can be very flexible to work in between 3D software and real-world environment. I expect this could be very helpful in the process of matte painting and composition. Moreover, as we pursuit more immersive experience, being able to copy or mapping the real world into virtual space is also appealing.
The plugin combines a set of different pre-trained machine learning models. Each of them is in charge of different functionality to construct the whole scene. The Detectron from Facebook Research is in charge of object detection which is able to provide the (x,y) coordinate and object name. The depth estimation model is in charge of (z) coordinate in the space. The 3D Bounding Box Estimation model output the rotation of objects.

Classes

Thesis