DeoVR App documentation

1. How to configure servers

The easiest way to stream video on demand (VoD) is to use progressive HTTP streaming which doesn't require any additional or special web server configuration. DeoVR player will download video header and based on this information will send HTTP Range requests to the web server in order to download necessary parts of the video file.

HTTPS is required to ensure compatibility with all Android versions.

Just upload your video files to the public folder of your Web Server and if you encoded your video right it should work without any additional server configuration.

There are 2 downsides of this method:

1. No seamless quality change based on a user internet speed
2. Users will be able to download your videos using a direct link.

2. Video encoding basics

There are quite a few different video and audio encodings available but not all of them you can use to play high-resolution VR video. All VR-ready PCs have hardware video decoding acceleration and it is a good idea to make use of it, more about it below. Set “Key Frame Distance” to 1-2 seconds, this will ensure good compromise between file size and smooth seeking.

There are simple rules you can follow to make sure that your customers will be able to watch your videos:

1. The most supported video codec today is H.264/MPEG-4 AVC, any VR-compatible PC can play it. Use it if you want maximum compatibility, you can use the same video file to stream your videos directly in users browsers. Select AAC audio encoding with a bitrate from 128k to 384k depending on your case and MP4 container format. It has limitations:
   1. Maximum resolution hardware decoders can handle is 4096x4096 @ 30 FPS or 4096x2048 @ 60 FPS, do not exceed these constraints to ensure smooth playback for your users.
   2. It requires more bitrate (internet bandwidth) to get the same visual quality than modern video codecs listed below.
2. More advanced video codec is H.265/HEVC, it provides the same visual quality at a lower bitrate than H.264 and NVidia GeForce GTX 10 Sires video cards can decode up to 8192x8192 @ 30 FPS with a built-in hardware video decoder and some mobile Exynos SoCs. Use the same audio settings and container format as for H.264 video codec. Limitations:
   1. Browsers do not support this codec this means that you can not use it for direct web-browser streaming.
   2. Windows 7 doesn’t support hardware accelerated video decoding for this codec.
   3. Takes much longer to encode with software video encoder compared to H.264
3. VP9 is an alternative video encoder from Google, it is very close to H.265 in terms of video quality for the same bitrate and it is open source software. Use Vorbis audio codec and WebM container format. Limitations:
   1. All Apple products don’t officially support it.
   2. @TODO: Need to test hardware decoding/Codecs on Win 7 and Win 10

If you want to ensure that all your users will be able to watch your videos, you should provide several streaming options to them. The absolute minimum would be:

1. Original H.265 video in the highest resolution and framerate for high-end users.
2. H.264 4k video option for the maximum compatibility.

DeoVR uses Rift_CV1 colors pace across all Oculus devices including Quest 2.
OpenXR specs say it is the preferred color space for standardised color across all Oculus HMDs:

1. Red (0.666, 0.334)
2. Green (0.238, 0.714)
3. Blue (0.139, 0.053)
4. White D75 (0.298, 0.318)

3. Intro to the integration

There are two ways to integrate DeoVR into the site.

A. Single video deeplink
A file .json with the description of one video is uploaded on the server. Then a button is added to the site. Once deeplink button is clicked, DeoVR launches and starts the video from this file. The video is played immediately.

B. Multiple videos selection

Called from DeoVR browser
Single .json file containing the list of videos is uploaded on the server. Then, in DeoVR “Internet browser” user calls the site (e.g. www.deovr.com) and gets access to the Selection Scene where they can choose a video from a list.

Called from a deeplink on the site
Single .json file containing the list of videos is uploaded on the server. Then a button with a deeplink to this file (e.g. “deovr://https://www.deovr.com/something.json”) is added to the site. By opening the site in a browser and clicking on the deeplink button, DeoVR launches and shows the user Selection Scene.

You can either implement just one integration, or both in the same time.

4. Single video deeplink formatting

IMPORTANT!
DeoVR saves settings for videos internally, title will be the title of the video, but if you want to send multiple videos with same title—use ID to distinct between them.

"title": "ExampleVideo"
"id": 555

1. Resolutions
Create a list of "videoSources". Set the URL and resolution for each video source.

"videoSources":
[
    {
        "resolution": 1080,
        "url": "C:/JsonExampleVideos/ExampleVideo_1080p.mp4"
    },
    {
        "resolution": 1440,
        "url": "C:/JsonExampleVideos/ExampleVideo_1440p.mp4"
    }
]

2. Video preview
You can add a video file which will be used to show the rewind of the file in the player.

"videoThumbnail":"https://yoursite.com/ExampleVideo_SeekLookUp.mp4"

The video should be in a low resolution as well as with low fps in order to save resources. In DeoVR for mobile devices this function may be unavailable.

3. Neccessary thumbnail and preview (optional) in case of playing from Selection Scene
In case of playing this video from the list, it is necessary to add the following lines:

"videoPreview":"https://yoursite.com/ExampleVideo_Preview.mp4"
"thumbnailUrl":"https://yoursite.com/ExampleVideo_Thumbnail.jpg"

The field ‘videoPreview’ contains the link to the video file, which is shown when moving the cursor to this video in the list. This field is not required.

The field ‘thumbnailUrl’ should contain the link to the file with the image shown in the list. This field is required in case of using the list.

4. Timestamps
Create a new list of "timeStamps". Each time stamp has time in seconds and a name value.

"timeStamps":
[
    {
        "ts": 15,
        "name": "Rabbit jumps"
    },
    {
        "ts": 30,
        "name": "Rabbit Sleeps"
    }
]

IMPORTANT!
For correct work of timeStamps you should also specify length of the video in seconds, otherwise unnecessary.

"videoLength": 60

5. Corrections
Declare "corrections" and set horizontal ("x") or vertical ("y") offset [min -7.5, max 7.5], brightness ("br"), contrast ("cont") or saturation ("sat") values [min -70, max 70].

"corrections":
    {
        "x": 5,
        "y": -5,
        "br": -10,
        "cont": 10,
        "sat": 20
    }

6. Other
"stereoMode" can be set to "sbs" for side by side stereoscopic layout, "tb" for top-bottom layout, "cuv" for custom UV layout (currently only used for Canon RF52 lens) or "off" for monoscopic videos.

"stereoMode": "tb"

You can set type of the mesh in "screenType" field if video data json.
Use the following values to adjust it:

"screenType": "sphere"

• "flat" - flat 2d video
• "dome" - 180 degrees equirect mesh
• "sphere" - 360 degrees equirect mesh
• "fisheye" - 180 degrees fisheye mesh
• "mkx200" - 200 degrees fisheye mesh
• "rf52" - 190 degrees Canon fisheye mesh

7. Full JSON example

{
  "encodings":[
    {
      "name":"h264",
      "videoSources":[
        {
          "resolution":1080,
          "url":"https://yoursite.com/ExampleVideo_1080p.mp4"
        },
        {
          "resolution":1440,
          "url":"https://yoursite.com/ExampleVideo_1440p.mp4"
        },
        {
          "resolution":1920,
          "url":"https://yoursite.com/ExampleVideo_1920p.mp4"
        },
        {
          "resolution":2160,
          "url":"https://yoursite.com/ExampleVideo_2160p.mp4"
        },
        {
          "resolution":2880,
          "url":"https://yoursite.com/ExampleVideo_2880p.mp4"
        },
        {
          "resolution":3360,
          "url":"https://yoursite.com/ExampleVideo_3360p.mp4"
        },
        {
          "resolution":3840,
          "url":"https://yoursite.com/ExampleVideo_3840p.mp4"
        }
      ]
    }
  ],
  "title":"ExampleVideo",
  "id":123,
  "videoLength":60,
  "is3d":true,
  "screenType":"sphere",
  "stereoMode":"tb",
  "skipIntro":0,
  "videoThumbnail":"https://yoursite.com/ExampleVideo_SeekLookUp.mp4",
  "videoPreview":"https://yoursite.com/ExampleVideo_Preview.mp4",
  "thumbnailUrl":"https://yoursite.com/ExampleVideo_image.jpg",
  "timeStamps":[
    {
      "ts":15,
      "name":"Wall"
    },
    {
      "ts":30,
      "name":"Window"
    },
    {
      "ts":45,
      "name":"Door"
    }
  ],
  "corrections":{
    "x":5,
    "y":-5,
    "br":-10,
    "cont":10,
    "sat":20
  }
}

IMPORTANT!
“is3d” should always be true, in case of false video will be forced monoscopic.

5. Multiple videos selection deeplink (Selection Scene)

Json file contains the a collection of single videos (see Single Video Deeplink). You can create multiple scenes which will be displayed as the tabs below the interface in Selection Scene.

Json file should contain at least one scene. Each scene corresponds with its own list of videos. An example of list of scenes format:

{
   "scenes":[
      {
         "name":"Trailers",
         "list":[
            … description of video as in single video deeplink …
         ]
      },
      {
         "name":"Full Videos",
         "list":[
            … description of video as in single video deeplink …
         ]
      }
   ],
   "authorized":"0"
}

In that case, two scenes named «Trailers» and «Full Videos» will be created.

An example of a full file with one scene and two videos:

{
  "scenes":[
    {
      "name":"Library",
      "list":[
        {
          "encodings":[
            {
              "name":"h264",
              "videoSources":[
                {
                  "resolution":1080,
                  "url":"https://yoursite.com/ExampleVideo_1080p.mp4"
                },
                {
                  "resolution":1440,
                  "url":"https://yoursite.com/ExampleVideo_1440p.mp4"
                }
              ]
            }
          ],
          "title":"ExampleVideo1",
          "screenType":"sphere",
          "stereoMode":"tb",
          "skipIntro":0,
          "videoThumbnail":"https://yoursite.com/ExampleVideo1_SeekLookUp.mp4",
          "videoPreview":"https://yoursite.com/ExampleVideo1_Preview.mp4",
          "thumbnailUrl":"https://yoursite.com/ExampleVideo1_image.jpg",
          "timeStamps":[
            {
              "ts":15,
              "name":"Wall"
            }
          ],
          "corrections":{
            "x":5,
            "y":-5,
            "br":-10,
            "cont":10,
            "sat":20
          },
          "is3d":true,
          "videoLength":60,
          "id":123
        },
        {
          "encodings":[
            {
              "name":"h264",
              "videoSources":[
                {
                  "resolution":1080,
                  "url":"https://yoursite.com/ExampleVideo2_1080p.mp4"
                },
                {
                  "resolution":1440,
                  "url":"https://yoursite.com/ExampleVideo2_1440p.mp4"
                }
              ]
            }
          ],
          "title":"ExampleVideo2",
          "screenType":"sphere",
          "stereoMode":"tb",
          "skipIntro":0,
          "videoThumbnail":"https://yoursite.com/ExampleVideo2_SeekLookUp.mp4",
          "videoPreview":"https://yoursite.com/ExampleVideo2_Preview.mp4",
          "thumbnailUrl":"https://yoursite.com/ExampleVideo2_image.jpg",
          "timeStamps":[
            {
              "ts":15,
              "name":"Wall"
            }
          ],
          "corrections":{
            "x":3,
            "y":-3,
            "br":-5,
            "cont":5,
            "sat":10
          },
          "is3d":true,
          "videoLength":65,
          "id":234
        }
      ]
    }
  ],
  "authorized":"0"
}

If the list of videos is too big, you can use shortened format.
In this case, instead of all fields required for the video to be played, only four fields are used with an addition of the field ‘video_url’, which contains Json to the full description of each video (see Single Video Deeplink).

The required fields of shortened format:

• thumbnailUrl - the address of the picture of the thumbnail;
• title – the title of the video;
• videoLength – the length of the video (in seconds);
• video_url – the address of Json with the description of the video

Example:

{
  "scenes":[
    {
      "name":"Library",
      "list":[
        {
          "title":"Play with a pretty dog",
          "videoLength":79,
          "thumbnailUrl":"https://deovr.com/s/images/feed/thumb1.png",
          "video_url":"https://deovr.com/deovr/video/id/1"
        },
        {
          "title":"Bikini car wash",
          "videoLength":242,
          "thumbnailUrl":"https://deovr.com/s/images/feed/thumb2.png",
          "video_url":"https://deovr.com/deovr/video/id/2"
        },
        {
          "title":"Date with a girl",
          "videoLength":401,
          "thumbnailUrl":"https:\/\/deovr.com\/s\/images\/feed\/thumb3.png",
          "video_url":"https://deovr.com/deovr/video/id/2"
        }
      ]
    }
  ]
}

Choosing of .json file name for DeoVR
If you want your site to be accessible from DeoVR Browser with the link as «http://www.yoursite.com», put file ‘deovr’ (without extension and quotes) into the root directory of the server.
When following a link containing only domain name, DeoVR will request the data at the address «http://www.yoursite.com/deovr».
In case calling any other URL, e.g. «http://www.yoursite.com/video/test» the request will be made at the same link without changes. The result of the server response should be a .json with a list of video description.

Authorization in DeoVR
In case of transferring login, the result of server response should also contain field “authorized” with the following values:
1 — user is successfully authorized;
0 — user without an account;
-1 — authorization error.
In case of authorization error, DeoVR shows the following message: «Invalid login or password!».
If authorization is not used, the field is not required or its value should equal to 0.

In case of authorization attempt, the current link with .json file (both the list and the video) will be opened through POST request method. Fields ‘login’ and ‘password’ will respective data entered by user. The result of the authorization should be reflected in the field ‘authorized’ of .json described earlier.
In case of using authorization, ‘login’ and ‘password’ will be transferred to the requests of obtaining scenes, as well as to the requests of video description.

6. Images Support

To provide user a feed of images (instead of videos), you must pass an image path (link to the image) in to the "path" variable instead of encoding classes. For StereoMode, ScreenType you can set screenType and stereoMode parameters or specify flags directly in the image file names.

Example:

{
  "scenes":[
    {
      "name":"Library",
      "list":[
        {
          "path":"https://yoursite.com/picture1_tb_360.jpg",
          "title":"ExamplePicture1",
          "screenType":"sphere",
          "stereoMode":"tb",
          "thumbnailUrl":"https://yoursite.com/thumbnail_picture1.jpg",
          "corrections":{
            "x":5,
            "y":-5,
            "br":-10,
            "cont":10,
            "sat":20
          },
          "is3d":true,
          "id":123
        },
        {
          "path":"https://yoursite.com/picture2_sbs_360.jpg",
          "title":"ExamplePicture2",
          "screenType":"sphere",
          "stereoMode":"sbs",
          "thumbnailUrl":"https://yoursite.com/thumbnail_picture2.jpg",
          "is3d":true,
          "videoLength":65,
          "id":234
        }
      ]
    }
  ],
  "authorized":"0"
}

7. Remote control

This guide will demonstrate you how remote control works. You can find a simple remote control client app with a source code here:

https://deovr.com/s/DeoRemoteControlTest.zip

The client is written in C# Windows Forms.

Preparing DeoVR:

1. Launch DeoVR app
2. Open settings menu in selection scene or file browser ("gear in right top corner")
3. Toggle switch "Enable remote control"
4. On Windows you will get notification from the firewall. Please confirm incoming connections
5. Start any video from selection scene or file browser.

Preparing Remote Control Client

1. Unpack and launch DeoRemoteControlTest app
2. If you started it on the same PC with DeoVR, click “Connect”, otherwise you should specify DeoVR IP address first
3. After connection established you will get message “Client connected”, after this you will start to receive data from DeoVR app
4. While you’re in the video, you will receive:
   • Path to the video file
   • Player state (could be play or pause)
   • Current video time
   • And current playback speed
5. You will be able to send to DeoVR:
   • New video path (new video are going to be opened immediately in DeoVR)
   • SeekTo value (will set new video time)
   • Playback speed (will change current playback speed)
   You can also send multiple parameters in the same packet.

Source code of this app is written in C# and is free to use. RemoteControlClient. cs could be used for comunication in separate program. It implements Connect, Disconnect and Send methods and OnConnected, OnDisconnected and OnDataReceived events. OnDataReceived would be executed each 1 second, when you are in the video player. OnDataReceived and Send works with RemoteApiData class which could be filled and sent or received back from DeoVR. You can check button handlers in TestForm. cs to see how to fills RemoteApiData.

Raw DeoVR remote control protocol description
Remote control client should connect to the device with running DeoVR on TCP port 23554.After the connection has been established, DeoVR will start sending a packet each one second. Remote client also must send a packet (empty or with json) to DeoVR each one secondfor pinging purposes.

Packet structure:

• Each packet starts with 4-bytes integer value with length of json data represented in UTF8 format.
• If length is zero, no video data is passed. Empty packets are used for ping purposes.
• If DeoVR won't receive any type of packet for more then 3 seconds it will close the connection.

Here's a structure of encoded json:

{
    "path":"D:/test.mp3",
    "duration":123.45,
    "currentTime":10.5,
    "playbackSpeed":1.0,
    "playerState":0
}

PlayerState is enumerator: Play = 0, Pause = 1. You can send packet with path, currentTime and playbackSpeed fields. DeoVR will automatically open new file/path, seek to special time or change playback speed. The only one condition, that you have to be inside video player in DeoVR.

8. Spatial Audio

DeoVR adds support for a spatial audio. Final VR video should be provided in .mkv container, with audio encoded in Opus codec. Only tbe8_2 format is supported at the moment. To learn more about creating spatial audio proceed to Spatial Workstation website. Select FB360 Matroshka (experimental) when exporting from Spatial Workstation.

• Deo_spatial_test_1_FB360_SBS_180.mkv
• Deo_spatial_test_2_FB360_SBS_180.mkv

To enable support of spatial audio on your website via DeoVR deeplink scheme add a separate “encodings_spatial” field in your JSON with encodings. Similar to “encodings” specify all other information but instead point to your .mkv files only with the spatial track.
Important: regular stereo track should be excluded from the file.

If you are playing a file with Path and not Encodings, the file should have the flag “-FB360” or “_FB360”. 
As of April 2021 switching between audio track manually is not yet supported.

Important when uploading video files with Spatial Audio.
The master file must be provided in the format: MP4 or MKV and contain a stereo track with two channels.
If you need to support Spatial Audio, the master file must be in MKV format and contain two tracks.
Track 1 should contain a track with Spatial Audio in Facebook 360 TBE 8.2 format.
Track 2 must be present and in 2-channel stereo format.

Example:

{
  "scenes":[
    {
      "name":"Library",
      "list":[
        {
          "encodings":[
            {
              "name":"h264",
              "videoSources":[
                {
                  "resolution":1080,
                  "url":"https://yoursite.com/ExampleVideo_1080p.mp4"
                },
                {
                  "resolution":1440,
                  "url":"https://yoursite.com/ExampleVideo_1440p.mp4"
                }
              ]
            }
          "encodings_spatial":[
            {
              "name":"h264",
              "videoSources":[
                {
                  "resolution":1080,
                  "url":"https://yoursite.com/SpatialExampleVideo_1080p.mkv"
                },
                {
                  "resolution":1440,
                  "url":"https://yoursite.com/SpatialExampleVideo_1440p.mkv"
                }
              ]
            }
          ],
          "title":"ExampleVideo1_Spatial",
          "screenType":"sphere",
          "stereoMode":"tb",
          "skipIntro":0,
          "videoThumbnail":"https://yoursite.com/ExampleVideo1_SeekLookUp.mp4",
          "videoPreview":"https://yoursite.com/ExampleVideo1_Preview.mp4",
          "thumbnailUrl":"https://yoursite.com/ExampleVideo1_image.jpg",
          "timeStamps":[
            {
              "ts":15,
              "name":"Wall"
            }
          ],
          "is3d":true,
          "videoLength":60,
          "id":123
        }
  ],
  "authorized":"0"
}

9. Naming convention

To play local files in the correct mode or add the appropriate flag to the file name. Currently the following flags are supported:

to switch StereoMode:

• _180
• _360
• _fisheye
• _fisheye190 \\Canon VR lens
• _rf52  \\Canon VR lens
• _mkx200
• _vrca220

to switch Screen Type:

• "LR" or "3DH" or "SBS"
• "TB" or "3DV" or "OverUnder"

For example:

Title_SBS_180.mp4 sets Side by Side & 180°
My_video_SBS_mkx200.mp4 sets Side by Side & 200°
Title_FB360_SBS_180.mkv sets Side by Side & 180 & Spatial sound

10. HLS, RTMP, RTSP

From now on you can easily stream from the Zcam K2 Pro camera to Oculus Quest and Quest 2 headsets with the help of Raspberry Pi. This brings the director's monitor view right into VR and you can adjust the scene before you press the record button. You might try recording and streaming at the same time.

Make sure to mount the Raspberry Pi SD image from the description to this video before you start. Use the original charger or appropriate power bank for the Raspberry Pi. Make sure to keep your Raspberry Pi away from routers or other WiFi devices.

Streaming works from a master camera, which shows footage in mono at 5Mbps with approximately 10 seconds latency. The master camera or camera A is the one with a screen and the record button.

Make sure you have DeoVR installed on your Quest headset. 

1. Turn the camera on.
2. In the camera menu go to Connect>Network>ETH. Set “Direct”
3. Turn on your Raspberry Pi. Connect the Raspberry Pi to the K2 Pro camera using the ethernet cable. 
4. The streaming will start automatically.
5. Connect your headset to the “RPI4_Camera_GW” WiFi network
6. In the DeoVR player app go to the browser and switch to http:// (click on https:// to switch it to http://). 
7. Type 10.0.0.1 (http://10.0.0.1) in the browser, click enter, then click on “Start Stream”.  You will experience a 10 sec delay.
8. If the camera is upside down, use the player settings to revert it. Go to Settings > Image tab and drag the “Rotation” slider to +/- 180 degrees.
9. Look at the image in VR and make any final adjustments to the scene.

This is RTSP streaming, the only option of K2 camera.

RTSP support is only available on the Windows platform.
RTMP not supported.
HLS supported.

If you want to open you own HLS stream in DeoVR, simply enter HLS stream url into DeoVR.
Or you can make json file named "deovr" with an entry to HLS stream. You can use this file to provide easy access to multiple HLS streams from DeoVR dashboard.
You can get more information about deovr json file in the article 5.

{
    "scenes": [
        {
            "name": "VR180 Samples",
            "list": [
                {
                    "path": "https://yoursite.com/Example.m3u8",
                    "title": " ExampleStream",
                    "screenType": " sphere",
                    "stereoMode": "sbs",
                    "thumbnailUrl": "https://yoursite.com/ExampleTumb.jpg",
                    "is3d": true,
                    "id": "01"
                }
            ]
        }
    ],
    "authorized": "0"
}

11. Supported type of meshes

• Flat 2D plane projection
• Equirectangular projection:
  • 180° FOV
  • 360° FOV
• Fisheye projection:
  • 180° FOV
  • 190° FOV
  • 200° FOV with MKX lens correction
  • 220° FOV with VRCA lens coorection 
  • Variable FOV (coming in 13.12)

Eye location:

• Side-by-Side
• Top-Bottom

12. Anaglyph mode

Anaglyph mode is a tool for VR creators that shows how one picture will be seen relative to another. It will greatly speed up the alignment process when creating a quick preview of your VR footage before you stitch together your video. It’s essential that the furthest point matches and that images are on the same horizontal line.
To use this tool, install this build on top of the DeoVR app.

13. Passthrough mode

1. Update to the latest version of the DeoVR app.
2. If using the Valve Index headset, make sure to enable Room View in SteamVR.
3. Find a suitable VR video on DeoVR with a black or plain background, or with the passthrough tag.
4. Open this video and go into the Player Settings (right-hand side panel). Go to the ‘Passthrough’ section at the bottom. Toggle ‘Enable passthrough’ on.
5. Now adjust the settings to make the background fade to reveal the real-world surroundings while the main action is still visible. Use Hue, Saturation and Brightness sliders to match the color you want to have transparent, after, increase the color range to start removing the color and colors similar to it. Use Falloff to soften the transition from visible to transparent pixels.

14. Alpha channel for Fisheye videos (13.8.1)