# UNHVD Unity Network Hardware Video Decoder Example of video and point cloud streaming with hardware decoding and custom [MLSP](https://github.com/bmegli/minimal-latency-streaming-protocol) protocol: - streaming video to UI element (RawImage) - streaming video to scene element (anything with texture) - streaming textured point clouds (Mesh) This project contains native library plugin. See [how-it-works](https://github.com/bmegli/unity-network-hardware-video-decoder/wiki/How-it-works) on wiki to understand the project. See [benchmarks](https://github.com/bmegli/unity-network-hardware-video-decoder/wiki/Benchmarks) on wiki for glass-to-glass latency. See [hardware-video-streaming](https://github.com/bmegli/hardware-video-streaming) for related projects. See videos to understand point cloud streaming features: | Point Cloud Streaming | Infrared Point Cloud Streaming | Point Cloud Streaming To UMPC | |-----------------------|-----------------------------------------|--------------------------------| | [![Hardware Accelerated Point Cloud Streaming](https://camo.githubusercontent.com/127f6aa745062b910252d3b1d7b05e1d9903f5253cbba31f5908bd5630417550/687474703a2f2f696d672e796f75747562652e636f6d2f76692f716e547868664e572d5f342f302e6a7067)](http://www.youtube.com/watch?v=qnTxhfNW-_4) | [![Hardware Accelerated Infrared Textured Point Cloud Streaming](https://camo.githubusercontent.com/79067547678eb7fbbae3df9f3585db3beda3c0acc77f7f3d7ef68026808ce571/687474703a2f2f696d672e796f75747562652e636f6d2f76692f7a56497576574d7a356d552f302e6a7067)](https://www.youtube.com/watch?v=zVIuvWMz5mU) | [![Realsense Wireless Point Cloud Streaming To UMPC](https://camo.githubusercontent.com/01f7816d79a1e2698510902cf4cbfcf1ffb1fd55dec351106ae6ad877025e13a/687474703a2f2f696d672e796f75747562652e636f6d2f76692f393532566b59506557334d2f302e6a7067)](https://www.youtube.com/watch?v=952VkYPeW3M) | ## Video sources Currently Unix-like platforms only. - [NHVE](https://github.com/bmegli/network-hardware-video-encoder) (dummy, procedurally generated video) - [RNHVE](https://github.com/bmegli/realsense-network-hardware-video-encoder) (Realsense camera streaming) ## Platforms Unix-like operating systems (e.g. Linux), [more info](https://github.com/bmegli/unity-network-hardware-video-decoder/wiki/Platforms). Tested on Ubuntu 18.04. ## Hardware Tested on Intel Kaby Lake. ### Video Intel VAAPI compatible hardware decoders (Quick Sync Video). It is likely that H.264 through VAAPI will work also on AMD and NVIDIA. [Other technologies](https://github.com/bmegli/unity-network-hardware-video-decoder/wiki/Hardware) may also work but were not tested. ### Depth/point clouds/textured point clouds Intel VAAPI HEVC Main10 compatible hardware decoders, at least Intel Apollo Lake. [Other technologies](https://github.com/bmegli/unity-network-hardware-video-decoder/wiki/Hardware) may also work but were not tested. [GPUPointCloudRenderer](https://github.com/bmegli/unity-network-hardware-video-decoder/wiki/How-it-works#gpupointcloudrenderer) has additional [requirements](https://github.com/bmegli/hardware-depth-unprojector#platforms) related to compute shaders. ## Dependencies All dependencies apart from FFmpeg are included as submodules, [more info](https://github.com/bmegli/unity-network-hardware-video-decoder/wiki/Dependencies). Works with system FFmpeg on Ubuntu 18.04 and doesn't on 16.04 (outdated FFmpeg and VAAPI ecosystem). ## Building Instructions Tested on Ubuntu 18.04.\ Requires Unity 2019.4 (LTS) ``` bash # update package repositories sudo apt-get update # get avcodec and avutil sudo apt-get install ffmpeg libavcodec-dev libavutil-dev # get compilers and make sudo apt-get install build-essential # get cmake - we need to specify libcurl4 for Ubuntu 18.04 dependencies problem sudo apt-get install libcurl4 cmake # get git sudo apt-get install git # clone the repository with *RECURSIVE* for submodules git clone --recursive https://github.com/bmegli/unity-network-hardware-video-decoder.git # build the plugin shared library cd unity-network-hardware-video-decoder cd PluginsSource cd unhvd-native mkdir build cd build cmake .. make # finally copy the native plugin library to Unity project cp libunhvd.so ../../../Assets/Plugins/x86_64/libunhvd.so ``` ## Testing Assuming you are using VAAPI device. 1. Open the project in Unity 2. Choose and enable GameObject 4. For Script define Configuration (`Port`, `Device`) 5. For troubleshooting use programs in `PluginsSource/unhvd-native/build` | Element | Video | Point Clouds | |------------------|---------------------------------------|-----------------------------------------| | GameObject | `Canvas` -> `CameraView` -> `RawImage` (UI)
`VideoQuad` (scene) | `PointCloud`
`GPUPointCloud` | | Script | `RawImageVideoRenderer`
`VideoRenderer` | `PointCloudRenderer`
`GPUPointCloudRenderer` | | Configuration | `Port` (network)
`Device` (acceleration>
script code | `Port` (network)
`Device` (acceleration)
script code | | Troubleshooting | `unhvd-frame-example` | `unhvd-cloud-example` | ### Sending side | Element | Video | Point Clouds | |------------------|---------------------------------------|-----------------------------------------| | Sending side | NHVE `nhve-stream-h264`
RNHVE `realsense-nhve-h264` | RNHVE `realsense-nhve-hevc`
RNHVE `realsense-nhve-depth-ir`
RNHVE `realsense-nhve-depth-color` | For a quick test you may use [NHVE](https://github.com/bmegli/network-hardware-video-encoder) procedurally generated H.264 video. If you have Realsense camera you may use [RNHVE](https://github.com/bmegli/realsense-network-hardware-video-encoder). #### Video ```bash # assuming you build NHVE port is 9766, VAAPI device is /dev/dri/renderD128 # in NHVE build directory ./nhve-stream-h264 127.0.0.1 9766 10 /dev/dri/renderD128 # if everything went well you will see 10 seconds video (moving through grayscale). # assuming you build RNHVE, port is 9766, VAAPI device is /dev/dri/renderD128 # in RNHVE build directory ./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 20 /dev/dri/renderD128 # if everything went well you will see 20 seconds video streamed from Realsense camera. ``` #### Point Clouds Assuming Realsense D435 camera and 848x480. ```bash # assuming you build RNHVE, port is 9768, VAAPI device is /dev/dri/renderD128 # in RNHVE build directory ./realsense-nhve-hevc 127.0.0.1 9768 depth 848 480 30 500 /dev/dri/renderD128 # for infrared textured point cloud ./realsense-nhve-depth-ir 127.0.0.1 9768 ir 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001 # for infrared rgb textured point cloud (D415/D455) ./realsense-nhve-depth-ir 127.0.0.1 9768 ir-rgb 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001 # for color textured point cloud, depth aligned ./realsense-nhve-depth-color 127.0.01 9768 depth 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001 # for color textured point cloud, color aligned, color intrinsics required ./realsense-nhve-depth-color 127.0.01 9768 color 848 480 848 480 30 500 /dev/dri/renderD128 8000000 1000000 0.0001 ``` If you are using different Realsense device/resolution you will have to configure camera intrinsics. See [point clouds configuration](https://github.com/bmegli/unity-network-hardware-video-decoder/wiki/Point-clouds-configuration) for the details. ## OS tweaks If you experience [stuttering with artifacts](https://github.com/bmegli/unity-network-hardware-video-decoder/issues/10#issuecomment-633255931) increase UDP buffer size. ```bash # here 10 x the the default value on my system sudo sh -c "echo 2129920 > /proc/sys/net/core/rmem_max" sudo sh -c "echo 2129920 > /proc/sys/net/core/rmem_default" ``` ## License Code in this repository and my dependencies are licensed under Mozilla Public License, v. 2.0 This is similiar to LGPL but more permissive: - you can use it as LGPL in prioprietrary software - unlike LGPL you may compile it statically with your code Like in LGPL, if you modify the code, you have to make your changes available. Making a github fork with your changes satisfies those requirements perfectly. Since you are linking to FFmpeg libraries consider also `avcodec` and `avutil` licensing.