Webcam Pipeline

The Webcam module in the HoloMIT SDK provides the basic 2D video conferencing capabilities by reading, encoding, transmitting, and decoding webcam feeds across online sessions. It is contained within its own dedicated library, Webcam.DLL.

Overview

The webcam subsystem is located under Assets\UserRepresentation\WebCam\ and utilizes a custom pipeline to stream video between users using FFmpeg for encoding and decoding. It supports both local (producer) and remote (consumer) streams with hardware acceleration when available.

Key Components

1. WebCamInitializer

The WebCamInitializer script is responsible for bootstrapping the webcam system when the application loads.

• Initialization: It registers the UserRepresentationType.WEBCAM.
• UI Registration: Configures the UI title ("2D Video"), description, and icon for the representation.
• Handler & Pipeline: Registers the WebCamRepresentationHandler and the WebCamPipeline into the SDK's central registries.

2. WebCamRepresentationHandler

Implementing IUserRepresentationHandler, this script is responsible for setting up the webcam for a specific player.

• When activated, it ensures the player's webcam GameObject is active.
• It attaches the WebCamPipeline component to the GameObject and invokes its Init() method with the correct parameters (e.g., whether the user is a producer or consumer).

3. WebCamPipeline

The WebCamPipeline is the core of the webcam stream processing, inheriting from BasePipeline. Its behavior changes depending on the stream source:

Local Stream (Producer)

When SourceType is Self, the pipeline acts as a producer:

• Reader: Initializes a WebCamReader to capture the local webcam texture.
• Encoder: If not in preview mode, it uses a VideoEncoder (supporting H264/H265) to encode the video. It can leverage hardware acceleration (CUDA, D3D11VA, DXVA2) depending on availability and priority.
• Writer: Uses MediaWriter to push the encoded video into the writerQueue for network transmission.

Remote Stream (Consumer)

When SourceType is Remote, the pipeline acts as a consumer:

• Reader: Uses MediaReader to fetch the incoming network stream into the decoderQueue.
• Decoder: Utilizes VideoDecoder (or AndroidVideoDecoder on mobile) to decode the FFmpeg frames.
• Preparer: Uses a VideoPreparer to stage the raw frames.
• Rendering: In the LateUpdate() method, the pipeline fetches the available video frame from the preparer, loads it into a Texture2D, and applies it to the PlayerHeadScreen material for display.

4. Shaders and Materials

The visual representation relies on a custom shader located in the Shaders folder.

• WebcamShader.shader: A custom shader used to render the 2D video texture onto the player's avatar.
• It features materials like Webcam.mat and WebcamGamma.mat to correctly process the gamma and color space of the incoming video feed.

Example of the 2D Video interface and representation in-game.