目录

Encoding with Preprocessing

Starting from API version 26.0.0, video encoders support preprocessing, including downsampling, cropping, and frame dropping.

Overview

Encoding preprocessing refers to the preprocessing capabilities executed by the video encoder before input frames enter the encoding pipeline. The primary encoder created via OH_VideoEncoder_CreatePrimaryWithPreproc supports the following three preprocessing features.

Feature Description Typical Scenario
Downsampling Scales high-resolution frames to a lower resolution before feeding them into the encoder. Low-resolution encoding output.
Cropping Extracts a specified rectangular region from the original frame for encoding. Region-of-interest (ROI) encoding and close-up views.
Frame dropping Selectively drops input frames based on the target frame rate. Bandwidth reduction and adaptive bitrate (ABR).

Mutual exclusion rule: Downsampling parameters and cropping parameters cannot be used simultaneously. Frame dropping can be combined with either downsampling or cropping.

[h2]When to Use

Scenario 1: Low-Resolution Encoding Output

Use downsampling to scale high-resolution input frames before encoding, producing a lower-resolution output stream.

Scenario 2: ROI Encoding

Scenario 3: ABR

[h2]Constraints

Basic Constraints

Item Description
Supported MIME types Same as the support scope of standard video encoders. Available via OH_AVCodec_GetCapability to query whether encoding of the specified MIME type is supported.
Creation method Must be created via OH_VideoEncoder_CreatePrimaryWithPreproc.
Data path Only Surface asynchronous mode is supported. Both Buffer mode and synchronous mode are not supported (returning AV_ERR_OPERATE_NOT_PERMIT).
Per-frame parameters The RegisterParameterCallback API is not supported (returning AV_ERR_OPERATE_NOT_PERMIT).

Downsampling Constraints

No. Constraint Rule
1 Paired configuration: The target width and target height for downsampling must be configured or updated together. If only one is specified, Configure or SetParameter returns AV_ERR_INVALID_VAL.
2 Disabling downsampling: Downsampling is disabled when both the target width and target height are set to valid zero values.
3 Valid range: Downsampling is enabled when both the target width and target height are within the supported ranges. You are advised to call OH_AVCapability_IsVideoSizeSupported to check whether the target size is supported.
4 Out-of-range values: If either the target width or target height is outside the supported range, Configure or SetParameter returns AV_ERR_INVALID_VAL.
5 Mutual exclusivity with cropping: Downsampling cannot be used together with the cropping parameters (CROP_LEFT, CROP_TOP, CROP_RIGHT, and CROP_BOTTOM). If both downsampling and cropping parameters are specified, AV_ERR_INVALID_VAL is returned.

Cropping Constraints

Coordinate system description:

No. Constraint Rule
1 Complete configuration: left, top, right, and bottom must all be configured together. If only some of these parameters are specified, AV_ERR_INVALID_VAL is returned.
2 Disabling cropping: Cropping is disabled when left, top, right, and bottom are all set to 0.
3 Valid range: Cropping is enabled when the width and height of the cropped region are within the supported ranges. You are advised to call OH_AVCapability_IsVideoSizeSupported to check whether the cropped size is supported.
4 Out-of-range values: If the width or height of the cropped region is outside the supported range, AV_ERR_INVALID_VAL is returned.
5 Mutual exclusivity with downsampling: Cropping cannot be used together with the downsampling parameters (DOWNSAMPLING_WIDTH and DOWNSAMPLING_HEIGHT). If both cropping and downsampling parameters are specified, AV_ERR_INVALID_VAL is returned.
6 Behavior: When cropping is enabled, the encoder encodes only the specified cropped region of each input frame. Content outside the cropped region is discarded.

Frame Dropping Constraints

No. Constraint Rule
1 Prerequisite: The caller must have set the original frame rate (OH_MD_KEY_FRAME_RATE).
2 Precision requirement: The value precision is retained to 2 decimal places (using rounding).
3 Value 0.00: The frame dropping feature is disabled.
4 Valid positive value: When set to a positive number greater than 0 and less than the original frame rate, frame dropping is performed at the configured frame rate.
5 Invalid value: When set to a negative number or a value greater than or equal to the original frame rate, AV_ERR_INVALID_VAL is returned.
6 Combinability: Can be used simultaneously with the downsampling parameter.
7 Combinability: Can be used simultaneously with the cropping parameter.

API Availability

The following table describes the availability of APIs for encoders created by calling OH_VideoEncoder_CreatePrimaryWithPreproc.

API availability Description
OH_VideoEncoder_RegisterCallback √ Supported.
OH_VideoEncoder_RegisterParameterCallback × Per-frame parameter configuration is not supported. Returns AV_ERR_OPERATE_NOT_PERMIT.
OH_VideoEncoder_PushInputParameter × Per-frame parameter configuration is not supported. Returns AV_ERR_OPERATE_NOT_PERMIT.
OH_VideoEncoder_Configure √ Supported. Preprocessing parameters can be configured.
OH_VideoEncoder_GetSurface √ Supported.
OH_VideoEncoder_Prepare √ Supported. Prepares internal resources.
OH_VideoEncoder_Start √ Supported.
OH_VideoEncoder_Stop √ Supported.
OH_VideoEncoder_Flush √ Supported.
OH_VideoEncoder_Reset √ Supported. Resets to the Initialized state.
OH_VideoEncoder_SetParameter √ Supported. Dynamically adjusts preprocessing and other parameters at runtime.
OH_VideoEncoder_NotifyEndOfStream √ Supported. Notifies the encoder of EOS information.
OH_VideoEncoder_FreeOutputBuffer √ Supported.
OH_VideoEncoder_GetInputDescription √ Supported. Preprocessing metadata is included.
OH_VideoEncoder_GetOutputDescription √ Supported.
OH_VideoEncoder_IsValid √ Supported.
OH_VideoEncoder_Destroy √ Supported. Destroys the encoder instance.
OH_VideoEncoder_PushInputData × Buffer mode is not supported.
OH_VideoEncoder_PushInputBuffer × Buffer mode is not supported.
OH_VideoEncoder_QueryInputBuffer × Synchronous mode is not supported.
OH_VideoEncoder_QueryOutputBuffer × Synchronous mode is not supported.

How to Develop

The workflow for encoding with pre-processing is the same as that for a regular encoder. The main differences are the encoder creation method, pre-processing parameter configuration, and dynamic parameter updates. This section describes only these differences. For the complete encoding workflow, see Surface Mode in video encoding.

[h2]Creating an Encoder with Preprocessing Support

Before creating an encoder with preprocessing support, you are advised to check whether the current encoder supports the required preprocessing features through the capability query API:

const char *mime = OH_AVCODEC_MIMETYPE_VIDEO_AVC;

OH_AVCapability *capability = OH_AVCodec_GetCapability(mime, true);
if (capability == nullptr) {
    return -1; // Failed to obtain capability, possibly due to an unsupported MIME type.
}

// Query whether downsampling preprocessing is supported.
bool supportDownsampling = OH_AVCapability_IsFeatureSupported(capability,
    VIDEO_ENCODER_PREPROC_DOWNSAMPLING);

// Query whether cropping preprocessing is supported.
bool supportCrop = OH_AVCapability_IsFeatureSupported(capability,
    VIDEO_ENCODER_PREPROC_CROP);

OH_AVCodec *encoder = nullptr;
OH_AVErrCode ret = OH_VideoEncoder_CreatePrimaryWithPreproc(mime, &encoder);
if (ret != AV_ERR_OK || encoder == nullptr) {
    // Handle the exception.
    return -1;
}

[h2]Registering Callbacks

Register callbacks by referring to step 3 (call OH_VideoEncoder_RegisterCallback() to register the callback functions) in Surface Mode of video encoding.

[h2]Setting Encoding and Preprocessing Parameters

Set encoding parameters by referring to step 5 (call OH_VideoEncoder_Configure() to configure the encoder) in Surface Mode of video encoding. The following focuses on the setting of basic parameters and preprocessing parameters.

OH_AVFormat *format = OH_AVFormat_Create();

// Set basic encoding parameters (mandatory).
OH_AVFormat_SetIntValue(format, OH_MD_KEY_WIDTH, 1920);         // Width of the input image.
OH_AVFormat_SetIntValue(format, OH_MD_KEY_HEIGHT, 1080);        // Height of the input image.
OH_AVFormat_SetDoubleValue(format, OH_MD_KEY_FRAME_RATE, 30.0); // Original frame rate (prerequisite for the frame dropping feature).

// Preprocessing parameters (configure as needed; verify feature support via IsFeatureSupported before configuration).
// Option A: Downsampling example.
// The following example encodes a 1920 x 1080 video after scaling it down to 640 x 360.
// Note: width and height must be specified in pairs.
if (supportDownsampling) {
    OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_PREPROC_DOWNSAMPLING_WIDTH, 640); // Downsampling target width.
    OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_PREPROC_DOWNSAMPLING_HEIGHT, 360); // Downsampling target height.
}

// Option B: Cropping example.
// The following example crops the central 1280 x 720 region from 1920 x 1080.
// Note: left/top/right/bottom must all appear together.
// Downsampling and cropping are mutually exclusive and cannot be used simultaneously.
// Example: left = 320, top = 180, right = 1599, bottom = 899; corresponding to: width = 1599 - 320 + 1 = 1280, height = 899 - 180 + 1 = 720.
// if (supportCrop) {
//     OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_PREPROC_CROP_LEFT, 320);
//     OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_PREPROC_CROP_TOP, 180);
//     OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_PREPROC_CROP_RIGHT, 1599);
//     OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_PREPROC_CROP_BOTTOM, 899);
// }

// Option C: Frame dropping example.
// The following example shows reducing from 30 fps to 15 fps (can be used alone or combined with Downsampling/Cropping).
// OH_AVFormat_SetDoubleValue(format, OH_MD_KEY_VIDEO_ENCODER_PREPROC_DROP_TO_FRAME_RATE, 15.0);

// Execute the configuration.
OH_AVErrCode ret = OH_VideoEncoder_Configure(encoder, format);
if (ret != AV_ERR_OK) {
    // Handle the error.
    OH_AVFormat_Destroy(format);
    return -1;
}
OH_AVFormat_Destroy(format);

[h2]Obtaining the Surface

// Key: The Surface can only be obtained through the primary encoder handle.
OHNativeWindow *window = nullptr;
OH_AVErrCode ret = OH_VideoEncoder_GetSurface(encoder, &window);
if (ret != AV_ERR_OK || window == nullptr) {
    // Handle the exception.
    return -1;
}

// Bind the window to a data source such as Camera or XComponent.
// For example: cameraManager->SetPreviewSurface(window)
//       nativeXComponent->SetSurface(window)

Key rules:

[h2]Preparing, Starting, and Writing Encoded Images

For details, see steps 7, 8, and 10 in Surface Mode of video encoding.

[h2](Optional) Dynamically Adjusting at Runtime

Dynamically modify preprocessing parameters at runtime via SetParameter:

// Dynamically adjust the target frame rate for frame dropping (for example, in response to network status changes).
// When network congestion occurs, aggressive frame dropping can be applied, e.g.: AdjustDropFrameRate(encoder, 10.0);
// When the network recovers, frame dropping can be canceled: AdjustDropFrameRate(encoder, 0.0);
void AdjustDropFrameRate(OH_AVCodec *enc, double targetFps)
{
    OH_AVFormat *param = OH_AVFormat_Create();
    if (targetFps > 0) {
        OH_AVFormat_SetDoubleValue(param,
            OH_MD_KEY_VIDEO_ENCODER_PREPROC_DROP_TO_FRAME_RATE, targetFps);
    } else {
        // Set to 0.00 to disable the frame dropping feature.
        OH_AVFormat_SetDoubleValue(param,
            OH_MD_KEY_VIDEO_ENCODER_PREPROC_DROP_TO_FRAME_RATE, 0.0);
    }
    OH_VideoEncoder_SetParameter(enc, param);
    OH_AVFormat_Destroy(param);
}

// Dynamically adjust the downsampling target dimensions.
void AdjustDownsampling(OH_AVCodec *enc, int newWidth, int newHeight)
{
    OH_AVFormat *param = OH_AVFormat_Create();
    OH_AVFormat_SetIntValue(param, OH_MD_KEY_VIDEO_ENCODER_PREPROC_DOWNSAMPLING_WIDTH, newWidth);
    OH_AVFormat_SetIntValue(param, OH_MD_KEY_VIDEO_ENCODER_PREPROC_DOWNSAMPLING_HEIGHT, newHeight);
    OH_VideoEncoder_SetParameter(enc, param);
    OH_AVFormat_Destroy(param);
}

Other encoder parameters can also be updated dynamically. For details, see step 9 in Surface Mode of video encoding.

[h2]Notifying End of Stream, Releasing Output Buffers, and Destroying the Encoder

The steps are the same as those for a regular encoder. For details, see steps 12, 13, and 17 in Surface Mode of video encoding.

Query via GetInputDescription

After preprocessing is enabled, you can query the actual input information and configuration parameters after preprocessing via GetInputDescription:

OH_AVFormat *inputDesc = OH_VideoEncoder_GetInputDescription(encoder);
if (inputDesc != nullptr) {
    int32_t originWidth = 0, originHeight = 0;
    OH_AVFormat_GetIntValue(inputDesc, OH_MD_KEY_WIDTH, &originWidth);
    OH_AVFormat_GetIntValue(inputDesc, OH_MD_KEY_HEIGHT, &originHeight);

    // Query the currently effective preprocessing configuration parameters.
    int32_t dsWidth = 0;
    if (OH_AVFormat_GetIntValue(inputDesc,
        OH_MD_KEY_VIDEO_ENCODER_PREPROC_DOWNSAMPLING_WIDTH, &dsWidth)) {
    }
}