Passthrough at DeoVR

Passthrough is a revolutionary new feature available in the latest update of the DeoVR app. Here’s our comprehensive guide to what passthrough is and how to get the best out of it in DeoVR. 

What Is Passthrough? 

Passthrough is an exciting feature that uses the AR function on your VR headset to make it feel like the action you see in VR is in your actual room in front of you. View this video in your VR headset through the DeoVR app for a quick example of how it works. 

Instructions: 

First, you need a VR headset with a front-facing camera, such as the Quest 2 or Valve Index. Then follow these instructions: 

  1. If you haven’t already, update to the latest version of the DeoVR app. Go to Global Settings -> Personalize -> Under 'Virtual Environments' select 'Passthrough'.
  2. If you use the Valve Index headset, make sure to enable Room View in SteamVR.
  3. Now find a suitable video on DeoVR with a black or plain background. You can get the list of passthrough videos by searching the category called “Passthrough”.
  4. Open your video and go into the Player Settings (right-hand side panel). You’ll see the ‘Passthrough’ section at the bottom. Toggle ‘Enable passthrough’ on.
  5. Now you can adjust the settings to make the plain background fade to show your real surroundings while the main action appears in front of you. For scenes with black backgrounds, try adjusting the opacity/Falloff levels at first before tweaking the other settings.
  6. If the scene has a single-colored background, use the chroma key wheel to choose the nearest color, then set the saturation and brightness bars to max, then move the Hue slider around. You might need to adjust the saturation and brightness levels until the chroma key box color matches the background colour of the video. Finally, adjust the Falloff/opacity level until only the background has disappeared. [See below for more info about the chroma key tool.]

Using the chroma key tool

Using the player, you can apply the chroma key algorithm to any video in real-time to make parts of a video transparent.

  • Use the chroma key wheel to choose the color closest to the background of the video
  • Use the sliders (Hue, Saturation and Brightness) to fine-tune the color so it closely matches the background and makes it transparent
  • The color range calculates the distance between pixel color and key color
  • A low value means that pixel color must closely match the key color
  • Higher values include more shades of key color. At the maximum value, it will make everything transparent
  • Falloff (previously 'opacity') defines how fast the transparency falls from opaque to transparent between a color that doesn't match the key color and the color closest to it
  • Adjust the color range and Falloff values until your image is clear and you see both your real-life surroundings and the model perfectly

Notes:

  • Keep in mind that black is not an ideal color for keying - all black/dark zones will be "erased" no matter where they are located on the image. For example, black hair, black clothes and eyes will become transparent
  • The algorithm is pretty simple for performance reasons and doesn't recognize objects at runtime
  • For best results, we recommend preparing the chroma key background in the video in the post-production/editing stage
  • Due to hardware limitations, passthrough is only supported on the DeoVR desktop app for the Valve Index, however, you can use passthrough on your standalone Quest or Pico headset using the DeoVR app or the DeoVR app sideloaded on your headset

Additionally, the v13.5 update of the DeoVR app has seen a number of improvements to the chroma key tool: 

  • The color range slider is now less sensitive
  • The color box is now visible when dragging sliders
  • Improved performance when using the color wheel
  • An icon has been added to thumbnails on the hub so users can easily see which videos have passthrough configured

Valve Users

If you're having trouble getting Passthrough to work on Valve Index, try this: double-press the button on the bottom left of the headset. This enables the camera to work during the scene. 

Useful Tips

Here are some ideas and suggestions that can help you get the most out of this amazing new function: 

  • It takes trial and error. Experiment with the opacity and saturation levels first to find out what works best for you and your surroundings, then try adjusting other settings.
  • Some users feel that white backgrounds (walls, bed sheets, etc) work better than dark backgrounds, as the white can act as a type of green screen.  
  • Be aware of certain elements that match the background or shadows in scenes with dark backgrounds as the passthrough function will show them as see-through.
  • The settings work differently on different videos - one level of opacity will work well for a particular scene or studio, but not for another, so you may need to adjust it each time.
  • Some users feel it works better in low-light situations, so try turning the lights off in your room when trying this, to decrease the visibility of imperfections. 

Use the video below to test this feature in your Quest or Pico headset. Turn on the passthrough option and use the following settings:

  • Hue: 33
  • Saturation: 85
  • Brightness: 62
  • Color range: 190
  • Fall off: 1000

How is Passthrough different on the Valve, Quests and Pico headsets?

In brief, very different passthrough technology is used for each VR headset. Let's take a closer look at each of the major VR headset models. 

Valve Index - Passthrough is achieved via the built-in RGB stereo camera mounted on the front of the headset panel. It provides a color stereo capture passthrough through a traditional two-camera setup and requires the user to activate a built-in feature on the SteamVR app called "Room View". We cannot force Passthrough to activate via the DeoVR app directly - it can only be activated through the SteamVR Room View software toggle button, or by double tapping the menu button on the controller. The process to achieve Passthrough on the Valve Index for our app is to ensure that the user first activates the room view, then toggles on the DeoVR app Passthrough button to enable color keying adjustments.

Quest - On the Meta Quest Pro, Passthrough is achieved via its built-in RGB stereo camera mounted on the front and assisted by three IR cameras (left, center and right). Meanwhile, the Quest 2 uses four IR cameras that stitch together to form the final stereo view. Both are capable of stereo-passthrough. The function to activate the hardware Passthrough is exposed on the API level and our awesome devs incorporated this into the DeoVR app. This enables us to toggle into Passthrough mode on-demand through our app while running it.

PICO - Passthrough for the PICO 4 is achieved via a single RGB camera that sits at the very center of the HMD front panel. It does provide color Passthrough but it is limited to a mono image feed. Whereas the PICO 3 uses the same tech as Quest 2 (four IR cameras) and does provide stereo passthrough. The difference between mono and stereo is largely related to the scale of a projected Passthrough scene in comparison to its projected 3D content. As mono does not provide any perceivable size (depth perception) the aspect ratio between the Passthrough feed and 3D content would be mismatched. Both PICO headsets come with API with on-demand Passthrough which means it can be accessed directly through the DeoVR app.

The only non-fully automatic Passthrough activation via the DeoVR app is the Valve Index headset. It requires Room View to be activated and this is what most people tend to forget about (although there is a clear note written next to the Passthrough switch in-app!).

Future Plans

We have many ideas for how to make this function even better. Here are some of them: 

  • Hardcoded settings for particular videos in the DeoVR app to make it easy to instantly enjoy passthrough.
  • Sharable settings for different videos. 

We’d love to know your ideas too. Join the discussion over on our forum!

Read also:
Great news for all you PICO headset owners: the DeoVR app is now available in the PICO store! The DeoVR app has been through all the necessary checks and is now available for all PICO users in th...
We’re delighted to announce that the DeoVR team will be attending CES 2023. We will be at the Venetian Expo in Las Vegas from the 5th of January to the 8th, showcasing the groundbreaking new fea...
Updated: 26/01/2023. DeoVR is now available on the Pico 4. Here’s our complete guide on how to get it up and running.  Via DeoVR from the Pico 4 store As of Thursday 26th January 2023...