A photo of Yves in front of a Virtual Production background created by VAPAI
Machine learning Narrative/storytelling Tool/Service


VAPAI is an Artificial Intelligence-powered tool that enables student and independent filmmakers with reduced budgets to access Virtual Production capabilities in a way that makes it accesible for everyone. With very simple inputs, storytellers can rapidly iterate and create detailed 3D scenes ready to be used for final-pixel imagery.


Yves Fogel


Sharon De La Cruz


Studying filmmaking in Uruguay, I noticed how we tend to see the capabilities of Hollywood as if it were a whole other world; we have no way of matching their budgets, so we limit ourselves to the stories we tell and how we tell them. One of the things that made me most excited about coming to study in the USA, and especially at ITP, was the access to state-of-the-art technologies unavailable in Uruguay. Virtual Production is advertised as the future of Hollywood, and there is hope that it will help democratize the art of filmmaking, allowing storytellers worldwide to share their stories. However, for this to be true, a major bottleneck needs to be addressed: the creation of 3D environments. Currently, there are many DIY solutions for Virtual Production sets, but the options to create environments easily and cheaply are very limited. You can download assets from marketplaces, but if we want our imagination not to be limited by our budget, they fall short very fast. We are currently living through an immense boom in the field of Artificial Intelligence (AI). New groundbreaking technologies are revealed every week. I have been in awe of Artificial Intelligence for the past five years, so I know there is a big potential for its use in the Virtual Production world. With this idea of bringing Hollywood’s capabilities to small markets, I created VAPAI. The current form of VAPAI is a workflow and an Unreal Engine plugin that allows anyone with just a few clicks, to create 3D scenes with the help of Artificial Intelligence.

A photo of Yves in front of a Virtual Production background created by VAPAI


My research started with a discussion with David Poyner and Sang Jin-Bae about AI-created images during Introduction to Virtual Production class. The consensus was that the tool was amazing, but we struggled to find its uses, still, I felt that there was something to explore in this intersection. I was intrigued by the potential of Virtual Production in emerging markets, so I got in touch with Alvaro Brechner, an Uruguayan filmmaker currently working in Spain. We had a long talk about the state and future of filmmaking. He confessed that he used Virtual Production just once and couldn’t believe he could work hours in a sunset scene. Still, his team wasn’t prepared for all the technical requirements to succeed in a Virtual Production studio. The main struggle was that every time they needed to change something in a scene, they had to call a specialist to manipulate the 3D environment. What was supposed to be an agile, iterative process ended up being very bureaucratic. If this happens to one of the most important Uruguayan filmmakers, what is left for the rest of us? My following research confirmed the hypothesis: there is a huge bottleneck in the creation of 3D scenes. The other leg of my research was on the AI side, exploring all the nooks and crannies of the web, looking for signs that the tools needed to create this plugin exist and how to use them best.

Technical Details

Currently, the plugin takes advantage of the capabilities of Stable Diffusion version 2.1. The user begins by generating an image using their Text-to-Image model. The next step is to predict depth by connecting VAPAI to Intel’s Dense Prediction Transformer. With this information, we segment the image into 2D layers. The final step is done using Stable Diffusion inpainting capabilities with their Image-to-Image module. All models can be used locally or served on the cloud by easily calling an API; recommended to run them on banana.dev. The plugin was created using Python for Unreal Engine.

An animated gif showing how the layers move according to the cameraAn image showing Tara and Yves playtesting VAPAIFinal output created in VAPAI plugin