README

AiMaskStreamApp requirements

Setup

• Install vcpkg to e.g. C:/vcpkg

vcpkg integrate install

• Install required packages

vcpkg install libuv:x64-windows uwebsockets-json:x64-windows zlib:x64-windows usockets:x64-windows

• Configure and generate the Visual Studio solution file
• If using CMake GUI, set the source folder to the root of the repository and the build folder to build, and set the toolchain file for cross-compiling to C:/vcpkg/scripts/buildsystems/vcpkg.cmake

mkdir build

• else:

cd build
cmake -DCMAKE_TOOLCHAIN_FILE=C:/vcpkg/scripts/buildsystems/vcpkg.cmake ..

Build

• Open the generated Visual Studio solution file and build the solution
• Built binaries will be in build/Release
• You might need to copy opencv_world346.dll from samples/external/opencv/bin to the binary directory to run the application

NVIDIA MAXINE VideoEffects SDK: API Source Code and Sample Applications

NVIDIA MAXINE Video Effects SDK enables AI-based visual effects that run with standard webcam input and can easily be integrated into video conference and content creation pipelines. The underlying deep learning models are optimized with NVIDIA AI using NVIDIA® TensorRT™ for high-performance inference, making it possible for developers to apply multiple effects in real-time applications.

The SDK has the following AI features:

• Virtual Background,which segments and masks the background areas in a video or image to enable AI-powered background removal, replacement, or blur.
• Artifact Reduction, which reduces compression artifacts from an encoded video while preserving the details of the original video.
• Super Resolution, which generates a detail-enhanced video with up to 4X high-quality scaling, while also reducing blocky/noisy artifacts and preserving textures and content. It is suitable for upscaling lossy content.
• Upscaler, which is a very fast and light-weight method to deliver up to 4X high-quality scaled video with an adjustable sharpening parameter. This feature can be optionally pipelined with the Artifact Reduction feature to enhance the scale while reducing the video artifacts.
• Video Noise Removal, which removes low-light camera noise from a webcam video while preserving the texture details.

The SDK provides several sample applications that demonstrate the features listed above in real time by using offline videos.

• AI Green Screen App, which is a sample app that demonstrates the Virtual background feature.
• VideoEffects App, which is a sample app that can invoke each of Artifact Reduction, Super Resolution or Upscaler features individually.
• UpscalePipeline App, which is a sample app that pipelines the Artifact Reduction feature with the Upscaler feature.
• DenoiseEffect App, which is a sample app that demonstrates the Video Noise Removal feature.

The input and output resolutions supported by the features of the SDK are listed below.

• The Artifact Reduction feature supports between 90p to 1080p as input resolutions.
• The Super Resolution feature supports between 90p to 2160p as input resolutions.
  • Super Resolution supports the following scaling factors: 4/3x (~1.33x), 1.5x, 2x, 3x and 4x.
  • 2160p input is only supported for the following scaling factors: 4/3x (~1.33x), 1.5x and 2x
  • The maximum output resolution for the Super Resolution feature is 4320p.
• The Upscaler feature supports any input resolution, and the following scaling factors: 4/3x (~1.33x), 1.5x, 2x, 3x and 4x.
• The Video Noise Removal feature supports between 80p to 1080p as input resolutions.
• The Virtual Background and Background Blur features require that an input image/video be at least 288 pixels high.

NVIDIA MAXINE VideoEffects SDK is distributed in the following parts:

• This open source repository that includes the SDK API and proxy linking source code, and sample applications and their dependency libraries.
• An installer hosted on NVIDIA Maxine End-user Redistributables page that installs the SDK DLLs, the models, and the SDK dependency libraries.

Please refer to the SDK System guide for configuring and integrating the SDK, compiling and running the sample applications. Please visit the NVIDIA MAXINE Video Effects SDK webpage for more information about the SDK.

System requirements

The SDK is supported on NVIDIA GPUs that are based on the NVIDIA® Turing™, Ampere™ or Ada™ architecture and have Tensor Cores.

• Windows OS supported: 64-bit Windows 10 or later
• Microsoft Visual Studio: 2017 (MSVC15.0) or later
• CMake: v3.12 or later
• NVIDIA Graphics Driver for Windows: 511.65 or later

NVIDIA MAXINE Branding Guidelines

If you integrate an NVIDIA MAXINE SDK within your product, please follow the required branding guidelines that are available here

Compiling the sample apps

Steps

The open source repository includes the source code to build the sample applications, and a proxy file NVVideoEffectsProxy.cpp to enable compilation without explicitly linking against the SDK DLL.

Note: To download the models and runtime dependencies required by the features, you need to run the SDK Installer.

1. In the root folder of the downloaded source code, start the CMake GUI and specify the source folder and a build folder for the binary files.

• For the source folder, ensure that the path ends in OSS.
• For the build folder, ensure that the path ends in OSS/build.

2. Use CMake to configure and generate the Visual Studio solution file.

• Click Configure.
• When prompted to confirm that CMake can create the build folder, click OK.
• Select Visual Studio for the generator and x64 for the platform.
• To complete configuring the Visual Studio solution file, click Finish.
• To generate the Visual Studio Solution file, click Generate.
• Verify that the build folder contains the NvVideoEffects_SDK.sln file.

3. Use Visual Studio to generate the application binary .exe file from the NvVideoEffects_SDK.sln file.

• In CMake, to open Visual Studio, click Open Project.
• In Visual Studio, select Build > Build Solution.

Documentation

Please refer to the online documentation guides -

• NVIDIA Video Effects SDK Programming Guide
• NVIDIA Video Effects SDK System Guide
• NvCVImage API Guide

PDF versions of these guides are also available at the following locations -

• NVIDIA Video Effects SDK Programming Guide
• NVIDIA Video Effects SDK System Guide
• NvCVImage API Guide