Passthrough, the next frontier of the Virtual Reality experience!

28.02.2023

Passthrough is a game-changing development and we at DeoVR are at the forefront of this exciting new tech. One of our expert devs, Faizal, has been exploring how it all works and looking at the possibilities for the future of VR. First, watch this useful video for a quick and easy example of how passthrough looks on the most popular VR headsets (go here to try the salsa passthrough video for yourself), then see below for his comprehensive look at the world of passthrough and VR.

What is VR passthrough?

Passthrough, in this context, refers to the capability of a VR headset to allow a camera overlay of the real world to be projected onto a user's VR view. The overlay is used as the background of VR content, with the primary VR content rendered in its foreground, creating a believable blend of your real-world surroundings and virtual reality.

What is the passthrough video experience?

The objective of the passthrough experience is to achieve a believable overlap of VR videos with the real world where, once such views are achieved, one would attain a significantly higher sense of immersion.

Through the use of passthrough, a user can actively include their very own real live environment, objects, and even themselves into their VR experience. Simply, add up a VR video experience to it and you'll get yourself the closest thing there is to a believable physical interaction with a VR video.

Check out the full list of passthrough videos on DeoVR, or click through to the salsa dancing video below to try it out for yourself:

How can you use DeoVR to view passthrough videos?

DeoVR has a nifty feature that prevents a selected color from being rendered in a video source, allowing the passthrough background to be made visible in the place of the removed color. Hence, the most ideal video to be used for passthrough requires it to be single color as its background. Most people that are familiar with the "green screen" effect or the term "color keying" should find this approach familiar. DeoVR currently incorporates an easy-to-tweak color wheel picker that allows you to quickly select the target color, and adjust its hue, brightness, saturation, color range, as well as its falloff level. This gives you the best control on the chroma keying adjustments where needed.

What headsets support passthrough?

The general rule of thumb is any headset that has video passthrough capability and runs on either SteamVR, PICO, or the Oculus/Meta platform. Unfortunately, the Windows Mixed Reality (WMR) platform does not provide the compatible passthrough system needed for it to work.

The following headsets have been tested to work well with the video passthrough feature of DeoVR:

Quest, Quest2, and Quest Pro
PICO3 and PICO4
Valve Index, VIVE Pro, and VIVE Pro 2

What are the differences in video passthrough experience to be expected from the various headsets available out there?

Each headset in the VR market currently has different passthrough traits made available to its users. Some are equipped with mono cameras, and some are equipped with stereo cameras as their video feed. Then there is also the question of whether it is color or monochrome(black/white) as their video feed. Any of these combinations can still give you the passthrough effect that you are looking for with a few acceptable tradeoffs.

A mono camera would be unable to project your real world with depth. In contrast, a stereo camera would provide a stereo projection (able to give depth perception) of your surrounding in VR. However, this would not impact your VR passthrough experience much, as you are often fixated on the foreground content more than the background (the subject) - in our case, a stereo video would often be the foreground. The extra added benefit of stereo projection here is that you can potentially introduce your actual hands, limbs, or any other objects into the VR scene without losing real-life scale and proportions. If you use a mono camera, you will see your hands or objects being either too big or too small when viewed on stereo VR foreground content.

When it comes to color or monochrome camera video feed, color video feed would be better, as it would blend well with the colored foreground VR content, making for a more believable experience. As with stereo vs mono, a monochrome camera, although lacking color, the contrast between the colored foreground and monochrome background would make the foreground even more immersive for some users. Most of the current color passthrough camera feeds suffer from low-resolution images that make the color dark and blurry under low light. This might be an advantage for some people, who then prefer monochrome over color.

Step-by-step instructions on using the DeoVR passthrough controller

Load up any videos with a single color as the background.
While the video is playing, activate the Player UI and click the setting icon on the right side.
Select the Passthrough setting (Eye icon)
You should now be presented with the Passthrough adjustments menu.
Toggle the Enable passthrough option to enable.
By default, the color selector is set to Black.
Move your pointer on the color wheel to select the target color (Hue) that closely matches your video background. If it is green, select any shade of green from the wheel. If it is white select white.
Now, in the center of the wheel, you will see a box showing the various Saturation or Brightness levels of the selected target color.
Move your pointer around the box inside the wheel to get to the closest match of the target color of the video's background.
Now, move the Color Range value up until you are satisfied with the blending of the video screen with the camera passthrough image.
Congratulations. Now, you should be able to experience passthrough video in VR.

Some tips on improving your video passthrough experience

If the transparent edges of the keyed video are slightly colored, try to drop the Brightness value by a couple of notches.
If your target video is not built for passthrough (not a single-colored background) You can still force a holographic effect on it to create a blend of your video background and the camera passthrough. All you need to do is to reduce the Falloff level until you have an agreeable mixture of a holographic blend. It won't be perfect, but it should give you a believable holographic video experience. You can increase or decrease the Brightness value accordingly to make the video image more prominent than your camera background feed.
Try to align the video content with your real-world furniture, body parts, or even objects for extra immersion
Activate the 6DOF and set it to the highest value for X, Y, and Z to 0.40 to make the subject in the video more rooted and not moving too much when you move your head. These are prominent effects that you might encounter when there are close-ups of subjects in a VR video scene.
If your headset supports hand tracking, activate the DeoVR hand tracking system to virtually touch your virtual video content (Quest, Quest 2, and Quest Pro only). This would give you extra immersion in the sensory touch department. What the eyes see, the mind feels.
If you have some physical toys that can bring extra sensory stimulation, try it out with your VR video passthrough experience. Just be careful with your real surrounding.
If you are using color passthrough via the Valve Index, you can help improve the quality of the camera passthrough by making your physical surrounding more brightly lit. The brighter, the better.
DeoVR will remember each video on its Passthrough settings that you had previously tweaked, so you don't have to memorize any values.

Important Reminder for SteamVR headsets users:

1) One time only: Please ensure that your camera is enabled: SteamVR-> settings-> camera = on and Room view Camera Mode is set to Translucent Passthrough

2) Extra step before enabling passthrough in DeoVR: Please activate the room view when you want to activate the passthrough mode in DeoVR.

You may double-tap the system button on your VR controller or head unit button to quickly toggle room mode off or on. Then follow the step-by-step instruction above.

What next for the passthrough experience?

We at DeoVR are constantly improving the way things work. There will always be a better way to achieve a better immersion and experience. And we are currently working on an improved solution to the current passthrough approach.

As a sneak peek, we can spill a little info on the upcoming improvements to the passthrough mechanic: currently, we are utilizing a chroma keying approach where we eliminate a color to make it transparent. This has its drawbacks. For example, if a subject you would like to remain visible has the same color as your targeted transparent color, it will also be made transparent. Clearly, this is undesirable.

The upcoming passthrough approach would be to use a technique called alpha packing. Alpha packing would see each image frame of the video to be accompanied by a black-and-white alpha reference image. The black-and-white image would be overlayed on top of the color video image frame. From here, any of the color images that fall inside the white overlaid areas will be displayed, and the ones that are in the black region will be made transparent. Here is a simple example of how alpha packing works:

We’d love to know your thoughts and ideas about this incredible feature. Join the discussion over on our friendly forum!