目录

ROI Video Encoding

Basic Concepts

Region of Interest (ROI) video encoding has been supported since API version 20. This feature is an advanced optimization technology extended based on the hardware H.264/H.265 encoding capability. Its core logic is to allocate more encoding resources to the specified key regions in the frame to achieve high-quality encoding, which ensures the clear presentation of content in ROI regions under limited bandwidth and significantly enhances the overall visual experience.

You can define ROI regions in the video frame (for example, human faces in live streaming, license plates in video security, etc.), and adjust the encoding quality difference between ROI and non-ROI regions by setting quality offset parameters, thus realizing the differentiated allocation of encoding resources.

When to Use

ROI video encoding applies to scenarios where the bit rate cannot meet the video quality requirements due to limited network bandwidth, and the key frame content (ROI regions) can be clearly defined. Examples include video calls, live video streaming, video security, and more.

Recommended ROI regions for each scenario are as follows:

Based on the encoding mode and ROI configuration method, the ROI encoding development examples are divided into two types: configuring ROI via NativeBuffer metadata (recommended) and configuring ROI via the encoding input parameter callback. You can choose based on your actual business and technical architecture.

Scenario Live Streaming/Video Call Video Recording Editing & Export/Content Publishing
ROI information producer Camera Camera App
ROI Information Retrieval Method Extracted from video frame NativeBuffer metadata Extracted from video frame NativeBuffer metadata Self-managed by the app
Direct Producer of Encoded Video Frames Graphics Graphics App
Encoding mode Surface mode Surface mode Buffer mode
ROI Parameter Configuration Method NativeBuffer metadata configuration (recommended) Encoding input parameter callback configuration Encoding input buffer callback configuration
Development Example Method 1: Configuring ROI via NativeBuffer Metadata (Recommended) Surface Mode: Encoding Input Parameter Callback Buffer Mode: Encoding Input Buffer Callback
  • Both live streaming and recording scenarios use Surface mode encoding. The only difference lies in the ROI configuration method. Method 1: Configuring ROI via NativeBuffer Metadata (Recommended) is simple to implement and naturally aligned with the encoded frames.
  • If the metadata of the encoder input buffer cannot be modified during frame processing (for example, when camera frames are directly sent to the encoder Surface), you can choose the encoding input parameter callback configuration method.

Constraints

Supported encoders: H.264 8-bit hardware encoding, H.265 8-bit hardware encoding, and H.265 10-bit hardware encoding

Supported bit rate control modes: variable bit rate (VBR), constant bit rate (CBR) and stable quality rate control (SQR)

Dependent on ROI detection and recognition capability: The encoder does not have the ROI detection and recognition capability, so the ROI encoding technology takes effect depending on the ROI information you input. You can design and implement the ROI recognition capability based on your business scenario, or obtain the ROI region through the face region information (metadata) natively provided by the system camera module to reduce development costs. For metadata configuration and retrieval, see Metadata (ArkTS) and Metadata (C/C++). For the development steps of extracting ROI information from the video frame NativeBuffer metadata, see Method 1: Configuring ROI via NativeBuffer Metadata (Recommended).

Parameter Requirements

ROI configuration parameters are ultimately delivered to the encoder in the form of strings. Starting from API version 26.0.0, the key-value pair format is supported, while the legacy numeric-only format remains compatible. All coordinate parameters and numeric parameters are integers.

When ROI data comes from the NativeBuffer metadata of a camera frame, you do not need to manually concatenate strings. You can use the OH_VideoMetadata_GetRoiCount, OH_VideoMetadata_ParseRoiString, and OH_VideoMetadata_AppendRoiString APIs to directly parse region information from the ROI metadata of the camera frame, set key-value pair parameters (such as OH_MD_KEY_VIDEO_METADATA_ROI_DELTA_QP and OH_MD_KEY_VIDEO_METADATA_ROI_SEM_LABEL), and automatically generate a configuration string that meets the encoder requirements.

String format:

Rectangle region definition:

An ROI is a rectangular region. Top, Left and Bottom, Right define the coordinates of the top-left and bottom-right corners of the ROI region in the image (as shown in figure 1).

Key-value pair parameters:

Parameter Description Value Range Mandatory Default Behavior
dqp Quantization parameter offset (DeltaQP) [-51, 51] No When not set, the encoder uses the default QP strategy (=-3).
slb Semantic Label 0 (unspecified) or 1 (face) No This parameter is only used for developers to distinguish ROI region types and does not affect encoding behavior.

Quantity and area constraints:

Figure 1: ROI coordinates and maximum allowed area ratio

Effectiveness Mechanism

The following two methods are supported for configuring ROI:

General effectiveness mechanism:

  1. ROI parameters support frame-by-frame delivery and take effect in real time. You do not need to query capabilities or configure global switches.
  2. If the system encoder does not support ROI encoding, the encoder ignores ROI parameters and performs normal encoding.
  3. The valid range of DeltaQp is [-51, 51]. The encoder overlays DeltaQp on the QP of the ROI region, and then limits the result to the range [minQp, maxQp] to obtain the final QP.
  4. When no ROI parameters are configured for a frame, if ROI encoding takes effect for the previous frame, the ROI information of the previous frame is reused for ROI encoding of the current frame. If normal encoding is used for the previous frame, normal encoding is performed for the current frame.
  5. If the ROI parameters configured for a frame fail to parse any valid ROI information, normal encoding will be performed.
  6. If multiple ROI regions overlap, only the first configured ROI region will take effect at the overlapping area in the order of configuration.

Mechanism unique to method 1: A maximum of 256 bytes of characters is supported, and the excess part is truncated.

Empty string handling differences:

Due to the differences in empty string processing, you are advised not to configure empty strings. If you need to disable ROI encoding for a frame, you can configure a string without position information (for example, "Clear" or ";").

Effectiveness priority: If ROI parameters are configured via both methods for a frame, only the ROI parameters delivered via the encoding input callback configuration method will take effect, regardless of whether valid ROI information can be parsed from them.

Method 2: Configuring ROI via the Encoding Input Parameter Callback

Method 2 uses the video encoding parameter OH_MD_KEY_VIDEO_ENCODER_ROI_PARAMS to configure ROI parameters in the encoding input callback. Depending on the encoding mode, it is divided into the following two modes:

[h2]Surface Mode: Encoding Input Parameter Callback

This method is also applicable to Surface mode. ROI information is extracted from the NativeBuffer metadata of the frame, and the configuration is completed through the encoding input parameter callback registered via OH_VideoEncoder_RegisterParameterCallback (as shown in figure 3).

The encoder triggers the parameter callback when receiving a video frame. Since the parameter callback does not include frame timestamp information, you need to use a synchronization queue (RoiQueue) keyed by PTS (timestamp) to pass the ROI data from the frame processing thread to the encoding callback in chronological order. In the callback, the ROI entry with the smallest PTS is popped from the queue and configured into the parameter format through OH_AVFormat_SetStringValue.

Figure 3: ROI configuration process via the encoding input parameter callback API

The development procedure is as follows:

  1. Link dynamic libraries in CMakeLists.txt.

    Same as step 1 in Method 1: Configuring ROI via NativeBuffer Metadata (Recommended).

  2. Extract ROI information from the NativeBuffer metadata of the video frame.

    Same as step 2 in Method 1: Configuring ROI via NativeBuffer Metadata (Recommended).

  3. Use the OH_VideoMetadata API to assemble the ROI configuration string.

    Same as step 3 in Method 1: Configuring ROI via NativeBuffer Metadata (Recommended).

  4. Implement the PTS synchronization queue and callback user data structure.

    Since the parameter callback does not include frame timestamp information, ROI data cannot be directly aligned with encoding frames. It is recommended to implement a synchronization queue keyed by PTS to pass the ROI data from the frame processing thread to the encoding callback in PTS order. The encoding callback accesses the queue through the queue pointer in the user data structure, popping the entry with the smallest PTS to obtain the ROI configuration string.

    CodecUserRoi is a nested structure defined inside the VideoEncoder class, containing a queue pointer for direct access by the callback.

    // User data structure for parameter callback configuration.
struct CodecUserRoi {
    VideoEncoder* vencoder = nullptr;
    RoiQueue* roiQueue = nullptr;
};

  5. Register the encoding input parameter callback.

    The callback must be registered after the encoder is created and before Configure; otherwise, it does not take effect.

    // Parameter callback configuration: register the parameter callback before Configure.
if (roiPathType_ == ROI_PATH_METADATA_CALLBACK) {
    userData_ = std::make_unique<CodecUserRoi>();
    if (userData_) {
        userData_->vencoder = this;
        userData_->roiQueue = &roiQueue_;
    }
    int32_t ret = OH_VideoEncoder_RegisterParameterCallback(encoder_, OnNeedInputParameter, userData_.get());
    CHECK_AND_RETURN_RET_LOG(ret == AV_ERR_OK, SAMPLE_ERR_ERROR,
                             "OH_VideoEncoder_RegisterParameterCallback failed, ret: %{public}d", ret);
    SAMPLE_LOGI("Parameter callback configuration: ROI parameter callback registered.");
}

    OH_VideoEncoder_RegisterParameterCallback must be called before OH_VideoEncoder_Configure. After the callback is registered, the OnNeedInputParameter callback is triggered each time the encoder receives a frame of input data.

  6. Configure ROI information in the encoding input parameter callback.

    When the encoder receives input data, the OnNeedInputParameter callback is triggered. In the callback, pop the ROI entry with the smallest PTS from the RoiQueue, and configure it into the parameter format through OH_AVFormat_SetStringValue. An empty string must also be configured into the encoding parameters to explicitly indicate that no ROI region is configured for the frame.

    static void OnNeedInputParameter(OH_AVCodec *codec, uint32_t index, OH_AVFormat *parameter, void *userData)
{
    auto *roiUserData = static_cast<VideoEncoder::CodecUserRoi *>(userData);
    if (!roiUserData || !roiUserData->roiQueue) {
        OH_VideoEncoder_PushInputParameter(codec, index);
        return;
    }

    std::string roiStr = roiUserData->roiQueue->Pop();
    OH_AVFormat_SetStringValue(parameter, OH_MD_KEY_VIDEO_ENCODER_ROI_PARAMS, roiStr.c_str());
    if (!roiStr.empty()) {
        SAMPLE_LOGI("ROI configured (parameter callback path): %{public}s", roiStr.c_str());
    }
    OH_VideoEncoder_PushInputParameter(codec, index);
}

  7. Pass the assembled ROI string together with the frame PTS to the RoiQueue of VideoEncoder.

    After extracting and assembling the ROI string in the frame processing thread, you need to push the ROI string into the RoiQueue of VideoEncoder, indexed by PTS. The RoiQueue is sorted by PTS to ensure that the encoding callback obtains ROI data in frame order, avoiding misalignment.

    int64_t pts = OH_NativeImage_GetTimestamp(nativeImage_);
if ((roiPathType_ == ROI_PATH_METADATA_CALLBACK || roiPathType_ == ROI_PATH_BUFFER_MODE) && onRoiStrAssembled_) {
    onRoiStrAssembled_(pts, assembledRoiStr);
}

    VideoEncoder pushes ROI data into the RoiQueue:

    // Parameter callback configuration: Push ROI entries into RoiQueue by PTS.
void VideoEncoder::PushRoiEntry(int64_t pts, const std::string &roiStr)
{
    roiQueue_.Push(pts, roiStr);
}

// Parameter callback configuration: Clear RoiQueue when ROI is disabled.
void VideoEncoder::ClearRoiQueue()
{
    roiQueue_.Clear();
}

The RoiQueue stores ROI entries sorted by PTS. The encoding callback retrieves the entry with the smallest PTS to ensure the order consistency between frames and ROI data. If the queue is empty, it waits for up to 3 ms before returning an empty string. The queue automatically cleans up stale entries older than 2 seconds to prevent unbounded growth. The ROI string (including an empty ROI string) must be enqueued for each frame. An empty ROI string is used to explicitly indicate that no ROI region is configured for the frame. When disabling ROI, call ClearRoiQueue to clear the queue.

[h2]Buffer Mode: Encoding Input Buffer Callback

In Buffer mode, video frames are sent to the encoder through OH_VideoEncoder_PushInputBuffer. Because Buffer mode has no encoder surface, the ROI string cannot be delivered through NativeBuffer metadata and must be configured through the encoding input callback: in the frame processing thread, push the ROI string together with the frame PTS into RoiQueue (the same PTS synchronization queue as in Surface Mode: Encoding Input Parameter Callback); when the encoder requests an input buffer, the OnNeedInputBuffer callback is triggered, in which the ROI string is retrieved from RoiQueue and set into the input buffer parameters (as shown in Figure 4).

Figure 4: ROI configuration process via the encoding input buffer callback API

The detailed development procedure is as follows:

  1. Link dynamic libraries in CMakeLists.txt.

    Same as step 1 in Method 1: Configuring ROI via NativeBuffer Metadata (Recommended).

  2. Extract ROI information from the NativeBuffer metadata of the video frame.

    Same as step 2 in Method 1: Configuring ROI via NativeBuffer Metadata (Recommended).

  3. Use the OH_VideoMetadata API to assemble the ROI configuration string.

    Same as step 3 in Method 1: Configuring ROI via NativeBuffer Metadata (Recommended).

  4. Pass the ROI string together with the frame PTS to RoiQueue.

    After extracting and assembling the ROI string in the frame processing thread, push the ROI string into RoiQueue indexed by PTS so that the encoding callback can retrieve it in frame order. The passing method is the same as step 7 in Surface Mode: Encoding Input Parameter Callback.

  5. Configure ROI information in the encoding input buffer callback.

    When the encoder requests an input buffer, the OnNeedInputBuffer callback is triggered. In the callback, retrieve the ROI string with the smallest PTS from RoiQueue, obtain the parameter format of the input buffer through OH_AVBuffer_GetParameter, set the ROI string using OH_AVFormat_SetStringValue, call OH_AVBuffer_SetParameter to write it back to the buffer so that the configuration takes effect, and finally enqueue the buffer for the consumer thread to fill in the frame pixel data.

    void CodecCallback::OnNeedInputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    if (userData == nullptr) {
        return;
    }
    CodecUserData *codecUserData = static_cast<CodecUserData *>(userData);
    // Buffer mode: Obtain the ROI string from RoiQueue and set it to the input Buffer parameters.
    if (codecUserData->roiPathType == ROI_PATH_BUFFER_MODE && codecUserData->roiQueue != nullptr) {
        std::string roiStr = codecUserData->roiQueue->Pop();
        OH_AVFormat *format = OH_AVBuffer_GetParameter(buffer);
        if (format != nullptr) {
            OH_AVFormat_SetStringValue(format, OH_MD_KEY_VIDEO_ENCODER_ROI_PARAMS, roiStr.c_str());
            OH_AVBuffer_SetParameter(buffer, format);
            OH_AVFormat_Destroy(format);
        }
    }
    std::unique_lock<std::mutex> lock(codecUserData->inputMutex);
    codecUserData->inputBufferInfoQueue.emplace(index, buffer);
    codecUserData->inputCond.notify_all();
}
    • After the ROI configuration is complete, you also need to fill the input buffer with frame pixel data and send it to the encoder through OH_VideoEncoder_PushInputBuffer. This is not elaborated here. For details, see the description of Buffer mode in asynchronous-mode video encoding.
    • OH_AVBuffer_GetParameter returns a copy of the parameters. You must call OH_AVBuffer_SetParameter to write it back for the ROI configuration to take effect, and call OH_AVFormat_Destroy to release it after use.
    • The PTS synchronization mechanism of RoiQueue and the clearing process when ROI is disabled are the same as described in Surface Mode: Encoding Input Parameter Callback.