Abstract

Studying filmmaking in Uruguay, I noticed how we tend to see the capabilities of Hollywood as if it were a whole other world; we have no way of matching their budgets, so we limit ourselves to the stories we tell and how we tell them. One of the things that made me most excited about coming to study in the USA, and especially at ITP, was the access to state-of-the-art technologies unavailable in Uruguay. Virtual Production is advertised as the future of Hollywood, and there is hope that it will help democratize the art of filmmaking, allowing storytellers worldwide to share their stories. However, for this to be true, a major bottleneck needs to be addressed: the creation of 3D environments. Currently, there are many DIY solutions for Virtual Production sets, but the options to create environments easily and cheaply are very limited. You can download assets from marketplaces, but if we want our imagination not to be limited by our budget, they fall short very fast. We are currently living through an immense boom in the field of Artificial Intelligence (AI). New groundbreaking technologies are revealed every week. I have been in awe of Artificial Intelligence for the past five years, so I know there is a big potential for its use in the Virtual Production world. With this idea of bringing Hollywood’s capabilities to small markets, I created VAPAI. The current form of VAPAI is a workflow and an Unreal Engine plugin that allows anyone with just a few clicks, to create 3D scenes with the help of Artificial Intelligence.

Research

My research started with a discussion with David Poyner and Sang Jin-Bae about AI-created images during Introduction to Virtual Production class. The consensus was that the tool was amazing, but we struggled to find its uses, still, I felt that there was something to explore in this intersection. I was intrigued by the potential of Virtual Production in emerging markets, so I got in touch with Alvaro Brechner, an Uruguayan filmmaker currently working in Spain. We had a long talk about the state and future of filmmaking. He confessed that he used Virtual Production just once and couldn’t believe he could work hours in a sunset scene. Still, his team wasn’t prepared for all the technical requirements to succeed in a Virtual Production studio. The main struggle was that every time they needed to change something in a scene, they had to call a specialist to manipulate the 3D environment. What was supposed to be an agile, iterative process ended up being very bureaucratic. If this happens to one of the most important Uruguayan filmmakers, what is left for the rest of us? My following research confirmed the hypothesis: there is a huge bottleneck in the creation of 3D scenes. The other leg of my research was on the AI side, exploring all the nooks and crannies of the web, looking for signs that the tools needed to create this plugin exist and how to use them best.

Technical Details

Currently, the plugin takes advantage of the capabilities of Stable Diffusion version 2.1. The user begins by generating an image using their Text-to-Image model. The next step is to predict depth by connecting VAPAI to Intel’s Dense Prediction Transformer. With this information, we segment the image into 2D layers. The final step is done using Stable Diffusion inpainting capabilities with their Image-to-Image module. All models can be used locally or served on the cloud by easily calling an API; recommended to run them on banana.dev. The plugin was created using Python for Unreal Engine.