Shooting in a VR Environment / by Kristen DiLiello

It's the future!  In addition to creating Visual Reality content, we can now also "traditionally" film people in virtual reality environments. 

What?? Virtual reality contains many innovative game and narrative experiences. The current problem is that not everyone owns a VR headset. They're still at a high price point and not everyone is into it (until they try it!). The best way to demonstrate a VR experience to someone without access to VR, is to traditionally film a person in VR. Below I will tell you how we (myself + Outpost VFX) accomplished it. 

Why??? Filming a composite of a person in a VR experience allows everyone else to witness the experience from a "normal" viewpoint.  Put it this way, if you only watch someone's point of view in VR, it's like watching footage from a go-pro haphazardly falling out of the sky, the player's point of view spins every direction on a static screen, which is disorienting and chaotic.  If you're watching someone in VR in reality they just look like a crazy person spinning around the room making erratic movements. Compositing the person into the environment is the best way to get a sense of what is going on, since from their POV the environment is static and we now can observe it in a normal way with them. 

How?? By mounting a virtual tracker (or an extra VR controller) onto your camera (any camera) it's possible to film someone while simultaneously showing the VR environment they're inside. This is called "mixed reality."  As a technophobe with an aversion to words like "virtual reality, mixed reality, sautéed reality, and upside-down and backwards reality" I have to say that the process (from a DP stand-point) is simple and a ton of fun.  I couldn't help but get giddily excited once I was operating a camera seeing and filming things that didn't exist in reality.  Once your virtual camera tracker and real camera are aligned, a feed goes to the computer, composites the subject (usually in front of a green screen), and sends the composited image back to your viewfinder allowing for the entire VR environment to be filmed. 

Here are some examples of the evolution of the process. Recently we worked with a game called RacketNX to show their game live at some VR conventions. In the future I am curious how this tool can be used in narrative environments or even as a pre-vis tool.  Hopefully this is just the beginning. 

Filming and live compositing at Unity Summit: 

 

This is another early test with a green screen where we filmed multiple VR experience being played: 

 

This is the first test we did with a webcam and no green screen: