目录

Temporally Scalable Video Coding

Basic Concepts

[h2]Introduction to Temporally Scalable Video Coding

Scalable video coding is an extended standard for video coding. SVC (short for Scalable Video Coding, an extension of the H.264 standard) and SHVC (short for Scalable High Efficiency Video Coding, an extension of the H.265 standard) are popular nowadays.

Scalable video coding allows conveyance of information structured in a hierarchical manner of spatial scalability, temporal scalability, and quality scalability.

Temporally scalable video coding refers to the process of encoding a video sequence into a set of layers that provide an increasing temporal resolution. The following figure shows the structure of a bitstream that contains four temporal layers and is constructed based on the reference relationship.

In scenarios where the channel condition is poor, frames can be dropped layer by layer in descending order (L3- > L2- > L1) to meet the changing requirements of transmission and decoding capabilities.

The figure below shows the new bitstream structure when the frames at L3 are dropped. The bitstream can be normally decoded while the frame rate is reduced by half. Dropping can occur at other layers in a similar way.

[h2]Structure of a Temporally Scalable Bitstream

A bitstream is organized by one or more Group of Pictures (GOPs). A GOP is a collection of consecutive pictures that can be independently decoded. It measures the distance between two I-frames (also named key frames).

A GOP can be further divided into one or more Temporal Group of Pictures (TGOPs), and each TGOP is composed by a base layer (BL) and one or more associated enhancement layers (ELs). For example, frame 0 to frame 7 in the foregoing four-layer temporally scalable bitstream form a TGOP.

[h2]How to Implement the Structure of a Temporally Scalable Bitstream

The temporally scalable bitstream structure is implemented by specifying reference frames, which are classified into the following types based on the duration of residence in a Decoded Picture Buffer (DPB):

Although a specific cross-frame reference structure can be implemented when there is more than one STR, the span supported by temporal scalability is limited due to an excessively short validity period. This problem does not exist when coming to the LTR, which also covers the cross-frame scenario of the STR. Therefore, the LTR is preferably used to implement the structure of a temporally scalable bitstream.

When to Use

You are advised to use temporal scalability in the following scenarios:

If your application does not involve dynamic adjustment of the temporal reference structure and the hierarchical structure is simple, you are advised to use global temporal scalability. Otherwise, enable LTR.

Constraints

Global Temporal Scalability

[h2]Available APIs

The global temporal scalability feature is suitable for low-latency real-time video communication scenarios. During encoder operation, you can switch between the scalable coding mode and the normal coding mode. After temporal layering is configured and takes effect on the encoder, the layering-related parameters cannot be dynamically modified. The development configuration parameters are as follows.

Parameter Description
OH_MD_KEY_VIDEO_ENCODER_ENABLE_TEMPORAL_SCALABILITY Enabled status of the global temporal scalability feature.
OH_MD_KEY_VIDEO_ENCODER_TEMPORAL_GOP_SIZE TGOP size of the global temporal scalability feature.
OH_MD_KEY_VIDEO_ENCODER_TEMPORAL_GOP_REFERENCE_MODE TGOP reference mode of the global temporal scalability feature.

Example 1: ADJACENT_REFERENCE in TGOP=4

Example 2: JUMP_REFERENCE in TGOP=4

Example 3: UNIFORMLY_SCALED_REFERENCE in TGOP = 4

[h2]How to Develop

You can learn the basic encoding process in Video Encoding. This section describes the differences from the basic video encoding process.

  1. When creating an encoder instance, check whether the video encoder supports the global temporal scalability feature.

    // 1.1 Obtain the video encoder capability instance. The following uses H.264 as an example.
OH_AVCapability *cap = OH_AVCodec_GetCapability(OH_AVCODEC_MIMETYPE_VIDEO_AVC, true);
// 1.2 Check whether the global temporal scalability feature is supported.
bool isSupported = OH_AVCapability_IsFeatureSupported(cap, VIDEO_ENCODER_TEMPORAL_SCALABILITY);

    If the feature is supported, it can be enabled.

    // Create a hardware encoder instance.
OH_AVCodec *videoEnc = OH_VideoEncoder_CreateByMime(OH_AVCODEC_MIMETYPE_VIDEO_AVC);

  2. In the configuration phase, configure the parameters related to the global temporal scalability feature.

    constexpr int32_t TGOP_SIZE = 3;
// 2.1 Create a temporary AVFormat used for configuration.
auto format = std::shared_ptr<OH_AVFormat>(OH_AVFormat_Create(), OH_AVFormat_Destroy);
if (format == nullptr) {
    // Handle exceptions.
}
// 2.2 Fill in the key-value pair of the parameter used to enable the feature.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODER_ENABLE_TEMPORAL_SCALABILITY, 1);
// 2.3 (Optional) Fill in the key-value pairs of the parameters that specify the TGOP size and reference mode.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODER_TEMPORAL_GOP_SIZE, TGOP_SIZE);
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODER_TEMPORAL_GOP_REFERENCE_MODE, ADJACENT_REFERENCE);
// 2.4 Configure the parameters.
int32_t ret = OH_VideoEncoder_Configure(videoEnc, format.get());
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  3. (Optional) During operation, you can dynamically enable or disable the temporal scalability capability.

    The sample code for dynamically enabling the temporal scalability capability is as follows:

    std::unique_lock<std::shared_mutex> lock(codecMutex);
// Reset the encoder videoEnc.
OH_AVErrCode resetRet = OH_VideoEncoder_Reset(videoEnc);
if (resetRet != AV_ERR_OK) {
    // Handle the exception.
}
inQueue.Flush();
outQueue.Flush();
// Reconfigure the encoder parameters.
auto format = std::shared_ptr<OH_AVFormat>(OH_AVFormat_Create(), OH_AVFormat_Destroy);
if (format == nullptr) {
    // Handle the exception.
}
// Configure the Temporal Scalable Coding feature parameters.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODER_ENABLE_TEMPORAL_SCALABILITY, 1);
constexpr int32_t TGOP_SIZE = 3;
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODER_TEMPORAL_GOP_SIZE, TGOP_SIZE);
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODER_TEMPORAL_GOP_REFERENCE_MODE, ADJACENT_REFERENCE);
OH_AVErrCode configRet = OH_VideoEncoder_Configure(videoEnc, format.get());
if (configRet != AV_ERR_OK) {
    // Handle the exception.
}
// The encoder is ready again.
OH_AVErrCode prepareRet = OH_VideoEncoder_Prepare(videoEnc);
if (prepareRet != AV_ERR_OK) {
    // Handle the exception.
}

    The sample code for dynamically disabling the temporal scalability capability is as follows:

    std::unique_lock<std::shared_mutex> lock(codecMutex);
// Reset the encoder videoEnc.
OH_AVErrCode resetRet = OH_VideoEncoder_Reset(videoEnc);
if (resetRet != AV_ERR_OK) {
    // Handle the exception.
}
inQueue.Flush();
outQueue.Flush();
// Reconfigure the encoder parameters.
auto format = std::shared_ptr<OH_AVFormat>(OH_AVFormat_Create(), OH_AVFormat_Destroy);
if (format == nullptr) {
    // Handle the exception.
}
// Configure the Temporal Scalable Coding feature parameters.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODER_ENABLE_TEMPORAL_SCALABILITY, 0);
OH_AVErrCode configRet = OH_VideoEncoder_Configure(videoEnc, format.get());
if (configRet != AV_ERR_OK) {
    // Handle the exception.
}
// The encoder is ready again.
OH_AVErrCode prepareRet = OH_VideoEncoder_Prepare(videoEnc);
if (prepareRet != AV_ERR_OK) {
    // Handle the exception.
}

  4. (Optional) During output rotation, you can obtain the temporal layer information corresponding to the bitstream.

    You can obtain the temporal layer information based on the configured TGOP parameters and the number of encoded frames.

    The sample code is as follows:

    constexpr int32_t TGOP_SIZE = 3;
uint32_t outPoc = 0;
// Obtain the relative position in the TGOP based on the number of valid frames in the output callback and determine the layer based on the configuration.
static void OnNewOutputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    struct OH_AVCodecBufferAttr attr;
    OH_AVErrCode ret = OH_AVBuffer_GetBufferAttr(buffer, &attr);
    if (ret != AV_ERR_OK) {
        // Handle exceptions.
    }
    // Set POC to 0 after the I-frame is refreshed.
    if (attr.flags & AVCODEC_BUFFER_FLAGS_SYNC_FRAME) {
        outPoc = 0;
    }
    // Skip this step only for the XPS output.
    if (attr.flags != AVCODEC_BUFFER_FLAGS_CODEC_DATA) {
        int32_t tGopInner = outPoc % TGOP_SIZE;
        if (tGopInner == 0) {
            // I-frames cannot be dropped in subsequent transmission and decoding processes.
        } else {
            // Non-I-frames can be dropped in subsequent transmission and decoding processes.
        }
        outPoc++;
    }
}

    Starting from API version 26.0.0, you can obtain temporal layer information through the OH_MD_KEY_VIDEO_ENCODER_TEMPORAL_LAYER_ID field, eliminating the need to calculate the temporal layer based on TGOP parameters and frame output information. A temporal layer number of 0 indicates the base layer, and 1 or above indicates an enhancement layer.

    The sample code is as follows:

    static int32_t GetTemporalLayerID(OH_AVBuffer *buffer)
{
    int32_t layerID = -1;
    OH_AVFormat *format = OH_AVBuffer_GetParameter(buffer);
    OH_AVFormat_GetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_TEMPORAL_LAYER_ID, &layerID);
    return layerID;
}

void SampleCallback::OnNewOutputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    if (userData == nullptr) {
        return;
    }
    // ...

    // Obtain the temporal layer information from AVBuffer.
    int32_t layerID = GetTemporalLayerID(buffer);
}

  5. (Optional) During output, use the temporal layer information obtained in step 4 to implement adaptive transmission or adaptive decoding based on the actual bandwidth or service scenario.

    Based on the temporally scalable bitstream and layer information, select a required layer for transmission, or carry the information to the peer for adaptive decoding.

LTR

[h2]Available APIs

The LTR feature provides a configuration of the frame-level reference relationship. It is suitable for flexible and complex temporally hierarchical structures.

Parameter Description
OH_MD_KEY_VIDEO_ENCODER_LTR_FRAME_COUNT Number of LTR frames.
OH_MD_KEY_VIDEO_ENCODER_PER_FRAME_MARK_LTR Marked as an LTR frame.
OH_MD_KEY_VIDEO_ENCODER_PER_FRAME_USE_LTR POC number of the LTR frame referenced by the current frame.

For example, to implement the four-layer temporally hierarchical structure described in Introduction to Temporally Scalable Video Coding, perform the following steps:

  1. In the configuration phase, set OH_MD_KEY_VIDEO_ENCODER_LTR_FRAME_COUNT to 5.

  2. In the input rotation of the running phase, configure the LTR parameters according to the following table, where **** means that no configuration is required.

    Configuration\POC 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    MARK_LTR 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1
    USE_LTR \ \ 0 \ 0 \ 4 \ 0 \ 8 \ 8 \ 12 \ 8

[h2]How to Develop

This section describes only the steps that are different from the basic encoding process. You can learn the basic encoding process in Video Encoding.

  1. When creating an encoder instance, check whether the video encoder supports the LTR feature.

    constexpr int32_t NEEDED_LTR_COUNT = 5;
bool isSupported = false;
int32_t supportedLTRCount = 0;
// 1.1 Obtain the encoder capability instance. The following uses H.264 as an example.
OH_AVCapability *cap = OH_AVCodec_GetCapability(OH_AVCODEC_MIMETYPE_VIDEO_AVC, true);
// 1.2 Check whether the LTR feature is supported.
isSupported = OH_AVCapability_IsFeatureSupported(cap, VIDEO_ENCODER_LONG_TERM_REFERENCE);
// 1.3 Determine the number of supported LTR frames.
if (isSupported) {
    OH_AVFormat *properties = OH_AVCapability_GetFeatureProperties(cap, VIDEO_ENCODER_LONG_TERM_REFERENCE);
    if (!OH_AVFormat_GetIntValue(properties, OH_FEATURE_PROPERTY_KEY_VIDEO_ENCODER_MAX_LTR_FRAME_COUNT, &supportedLTRCount)) {
        // Handle exceptions.
    }
    OH_AVFormat_Destroy(properties);
    // 1.4 Check whether the number of supported LTR frames meets the structure requirements.
    isSupported = supportedLTRCount >= NEEDED_LTR_COUNT;
}

    If the LTR feature is supported and the number of LTR frames meets the requirements, the feature can be enabled.

  2. Register the frame channel callback functions.

    The following is an example of the configuration in buffer input mode:

    // 2.1 Implement the OH_AVCodecOnNeedInputBuffer callback function.
static void OnNeedInputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    // The index of the input frame buffer is sent to InIndexQueue.
    // The input frame data (specified by buffer) is sent to InBufferQueue.
    // Perform data processing. For details, see:
    // - Write the stream to encode.
    // - Notify the encoder of EOS.
    // - Write the frame parameter.
    OH_AVFormat *format = OH_AVBuffer_GetParameter(buffer);
    if (format == nullptr) {
        // Handle exceptions.
    }
    OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_PER_FRAME_MARK_LTR, 1);
    OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_PER_FRAME_USE_LTR, 4);
    OH_AVBuffer_SetParameter(buffer, format);
    OH_AVFormat_Destroy(format);
    // Notify the encoder that the buffer input is complete.
    OH_VideoEncoder_PushInputBuffer(codec, index);
}

// 2.2 Implement the OH_AVCodecOnNewOutputBuffer callback function.
static void OnNewOutputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    // The index of the output frame buffer is sent to outIndexQueue.
    // The encoded frame data (specified by buffer) is sent to outBufferQueue.
    // Perform data processing. For details, see:
    // - Release the encoded frame.
    // - Record POC and the enabled status of LTR.
}

// 2.3 Register the callback functions.
OH_AVCodecCallback cb;
cb.onNeedInputBuffer = OnNeedInputBuffer;
cb.onNewOutputBuffer = OnNewOutputBuffer;
OH_VideoEncoder_RegisterCallback(videoEnc, cb, nullptr);

    The following is an example of the configuration in surface input mode:

    // 2.1 Implement the OH_VideoEncoder_OnNeedInputParameter callback function.
static void OnNeedInputParameter(OH_AVCodec *codec, uint32_t index, OH_AVFormat *parameter, void *userData)
{
    if (parameter == nullptr) {
        // Exception handling.
    }
    // The index of the input frame buffer is sent to InIndexQueue.
    // The input frame data (specified by avformat) is sent to InFormatQueue.
    // Perform data processing. For details, see:
    // - Write the stream to encode.
    // - Notify the encoder of EOS.
    // - Write the frame parameter.
    OH_AVFormat_SetIntValue(parameter, OH_MD_KEY_VIDEO_ENCODER_PER_FRAME_MARK_LTR, 1);
    OH_AVFormat_SetIntValue(parameter, OH_MD_KEY_VIDEO_ENCODER_PER_FRAME_USE_LTR, 4);
    // Notify the encoder that the frame input is complete.
    OH_VideoEncoder_PushInputParameter(codec, index);
}

// 2.2 Implement the OH_AVCodecOnNewOutputBuffer callback function.
static void OnNewOutputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    // The index of the output frame buffer is sent to outIndexQueue.
    // The encoded frame data (specified by buffer) is sent to outBufferQueue.
    // Perform data processing. For details, see:
    // - Release the encoded frame.
    // - Record POC and the enabled status of LTR.
}

// 2.3 Register the callback functions.
OH_AVCodecCallback cb;
cb.onNewOutputBuffer = OnNewOutputBuffer;
OH_VideoEncoder_RegisterCallback(videoEnc, cb, nullptr);
// 2.4 Register the frame channel callback functions.
OH_VideoEncoder_OnNeedInputParameter inParaCb = OnNeedInputParameter;
OH_VideoEncoder_RegisterParameterCallback(videoEnc, inParaCb, nullptr);

  3. In the configuration phase, configure the maximum number of LTR frames.

    constexpr int32_t NEEDED_LTR_COUNT = 5;
// 3.1 Create a temporary AVFormat used for configuration.
auto format = std::shared_ptr<OH_AVFormat>(OH_AVFormat_Create(), OH_AVFormat_Destroy);
if (format == nullptr) {
    // Handle exceptions.
}
// 3.2 Fill in the key-value pair of the parameter that specifies the number of LTR frames.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODER_LTR_FRAME_COUNT, NEEDED_LTR_COUNT);
// 3.3 Configure the parameters.
int32_t ret = OH_VideoEncoder_Configure(videoEnc, format.get());
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  4. (Optional) During output rotation, obtain the temporal layer information corresponding to the bitstream.

    This procedure is the same as that described in the global temporal scalability feature.

    The LTR parameters are configured in the input rotation. You can also record the LTR parameters in the input rotation and find the corresponding input parameters in the output rotation.

  5. (Optional) During output rotation, use the temporal layer information obtained for adaptive transmission or decoding.

    This procedure is the same as that described in the global temporal scalability feature.