2D Video - Video Codecs

The Video Codecs module within the HoloMIT SDK handles the compression, decompression, and color space conversion of video frames across the platform. Built on top of FFmpeg.AutoGen, it leverages FFmpeg's robust capabilities, including hardware acceleration, to ensure efficient real-time video streaming.

Location

The source code for the video codecs can be found at: Assets\HoloVideo\Scripts\Workers\Codecs\.

Key Components

1. VideoEncoder

The VideoEncoder script is responsible for taking raw video frames and compressing them for network transmission. Inheriting from BaseWorker, it operates asynchronously using thread-safe queues (inVideoQueue and outVideoQueue).

• Initialization: Configured with a Setup struct that defines the resolution, framerate, GOP size, compression method (CRF or QP), pixel format, and hardware device type.
• Hardware Acceleration: Supports hardware encoders like NVENC for H.264 (h264_nvenc) and HEVC (hevc_nvenc) when using CUDA.
• Pipeline: It dequeues a raw RGB frame, converts it to a YUV format (like YUV420P) using VideoFilter, encodes it via FFmpeg (avcodec_send_frame), and enqueues the compressed packet to the output queue.
• Metrics: Integrated with the SDK's metrics system to report FPS, processing time, and packet sizes (via Prometheus or local files).

2. VideoDecoder

The VideoDecoder complements the encoder by decompressing incoming network packets back into raw frames. It also inherits from BaseWorker.

• Initialization: Sets up the FFmpeg codec context based on the stream's metadata (codec ID, pixel format, etc.).
• Hardware Acceleration: Highly versatile, supporting numerous hardware decoding APIs including CUDA (cuvid), DXVA2, D3D11VA, VDPAU, and Vulkan.
• Pipeline: Receives an encoded packet, passes it to the decoder (avcodec_send_packet), retrieves the decoded YUV AVFrame, and finally converts it back to an RGB24 format using FFmpeg's sws_scale. If hardware acceleration is used, it handles the memory transfer from the GPU to the CPU (av_hwframe_transfer_data).

3. AndroidVideoDecoder

Due to platform-specific constraints and optimizations, Android uses a dedicated decoder.

• JNI Integration: Instead of relying solely on FFmpeg, this decoder communicates with an Android Java plugin (com.example.decodertest.DecoderTest) via AndroidJavaObject to decode the frame natively on the device.
• Color Space Conversion: The Android decoder manually converts the resulting NV12 formatted YUV data into an RGBA byte array through a custom TransformNV12toRGBA function, bypassing FFmpeg's scaler for performance reasons.

4. VideoFilter

A utility class that simplifies the usage of FFmpeg's sws_scale.

• Used primarily by the VideoEncoder (and potentially decoders) to perform fast image scaling and pixel format conversions (e.g., RGB24 to YUV420P) before encoding or after decoding.