目录

Video Encoding

Video encoding is an important process in multimedia processing. It compresses uncompressed video data into video bitstreams to reduce the size of raw video data for storage or transmission. Video encoding supports synchronous mode and asynchronous mode. The main difference between the two modes lies in whether buffers are obtained synchronously or asynchronously. You can choose the API calling mode that best suits your service requirements.

This document mainly describes asynchronous video encoding. For synchronous video encoding, see Synchronous Video Encoding. Based on how data is input before encoding, the encoder supports surface mode and buffer mode for different use cases.

Difference Surface Mode Buffer Mode
Configuration Before calling OH_VideoEncoder_Prepare, you must call OH_VideoEncoder_GetSurface to obtain OHNativeWindow for transferring video data. -
Data input Input frames are obtained through OHNativeWindow. Data is usually written directly by a producer module, such as the camera. Shared memory buffer information is obtained through the OnNeedInputBuffer callback, and data is sent by calling OH_VideoEncoder_PushInputBuffer.
End of input You must call OH_VideoEncoder_NotifyEndOfStream to notify the encoder that input is complete. When encoding the last frame, set the flags field of the input buffer to AVCODEC_BUFFER_FLAGS_EOS to notify the encoder that input is complete.

For details about the video encoding formats supported by AVCodec, see Video Encoding.

For details about the implementation, see Samples.

State Machine Interaction

  1. Initialized state.

    • When an encoder instance is initially created, the encoder enters the Initialized state.

    • When OH_VideoEncoder_Reset is called in any state, the encoder can return to the Initialized state.

  2. Configured state.

    • In the Initialized state, call OH_VideoEncoder_Configure to configure the encoder. After the configuration is successful, the encoder enters the Configured state.

  3. Prepared state.

    • In the Configured state, call OH_VideoEncoder_Prepare to enter the Prepared state.

    • In the Executing state, call OH_VideoEncoder_Stop to return to the Prepared state.

  4. Executing state.

    • In the Prepared state, call OH_VideoEncoder_Start to enter the Executing state.

    • The Executing state has three substates: Running, Flushed, and End-of-Stream.

      • Running: Call OH_VideoEncoder_Start to enter the Running substate.

      • Flushed: Call OH_VideoEncoder_Flush to enter the Flushed substate.

      • End-of-Stream: When the encoder receives an input buffer whose flags field is AVCODEC_BUFFER_FLAGS_EOS in OH_AVCodecBufferFlags, or when OH_VideoEncoder_NotifyEndOfStream is called, the encoder enters the End-of-Stream substate. In this state, the encoder does not accept new inputs, but continues to generate outputs until the tail frame is output.

  5. Error state.

    • In rare cases, the encoder enters the Error state when an exception occurs. An API returns an error code or an exception is reported through the OH_AVCodecOnError callback.

    • In the Error state, call OH_VideoEncoder_Reset to return to the Initialized state, or call OH_VideoEncoder_Destroy to enter the final Released state.

  6. Released state.

    • After using the encoder, you must call OH_VideoEncoder_Destroy to destroy the encoder instance and switch the encoder to the Released state.

Figure 1 State machine interaction

How to Develop

Read native_avcodec_videoencoder.h for the API reference.

Figure 2 Video encoding call relationship

[h2]Linking the Dynamic Libraries in the CMake Script

target_link_libraries(sample PUBLIC libnative_media_codecbase.so)
target_link_libraries(sample PUBLIC libnative_media_core.so)
target_link_libraries(sample PUBLIC libnative_media_venc.so)

The word sample in the preceding code snippet is only an example. Use the actual project directory name.

[h2]Defining the Basic Structure

The sample code provided in this section adheres to the C++17 standard and is for reference only. You can define your own buffer objects by referring to it.

  1. Add the header files.

    #include <condition_variable>
#include <memory>
#include <mutex>
#include <queue>
#include <shared_mutex>

  2. Define the information about the encoder callback buffer.

    struct CodecBufferInfo {
    CodecBufferInfo(uint32_t index, OH_AVBuffer *buffer): index(index), buffer(buffer), isValid(true) {}
    CodecBufferInfo(uint32_t index, OH_AVFormat *parameter): index(index), parameter(parameter), isValid(true) {}
    // Callback buffer.
    OH_AVBuffer *buffer = nullptr;
    // In surface mode, pass the frame-specific parameter of the callback, which can be used only after the frame-specific parameter callback function is registered.
    OH_AVFormat *parameter = nullptr;
    // Index of the callback buffer.
    uint32_t index = 0;
    // Check whether the current buffer information is valid.
    bool isValid = true;
};

  3. Define the input and output queue for encoding.

    class CodecBufferQueue {
public:
    // Pass the callback buffer information to the queue.
    void Enqueue(const std::shared_ptr<CodecBufferInfo> bufferInfo)
    {
        std::unique_lock<std::mutex> lock(mutex_);
        bufferQueue_.push(bufferInfo);
        cond_.notify_all();
    }

    // Obtain the information about the callback buffer.
    std::shared_ptr<CodecBufferInfo> Dequeue(int32_t timeoutMs = 1000)
    {
        std::unique_lock<std::mutex> lock(mutex_);
        (void)cond_.wait_for(lock, std::chrono::milliseconds(timeoutMs), [this]() { return !bufferQueue_.empty(); });
        if (bufferQueue_.empty()) {
            return nullptr;
        }
        std::shared_ptr<CodecBufferInfo> bufferInfo = bufferQueue_.front();
        bufferQueue_.pop();
        return bufferInfo;
    }

    // Clear the queue. The previous callback buffer becomes unavailable.
    void Flush()
    {
        std::unique_lock<std::mutex> lock(mutex_);
        while (!bufferQueue_.empty()) {
            std::shared_ptr<CodecBufferInfo> bufferInfo = bufferQueue_.front();
            // After the flush, stop, reset, and destroy operations are performed, the previous callback buffer information is invalid.
            bufferInfo->isValid = false;
            bufferQueue_.pop();
        }
    }

private:
    std::mutex mutex_;
    std::condition_variable cond_;
    std::queue<std::shared_ptr<CodecBufferInfo>> bufferQueue_;
};

  4. Configure global variables.

    These global variables are for reference only. They can be encapsulated into an object based on service requirements.

    // Video frame width.
int32_t width = 320;
// Video frame height.
int32_t height = 240;
// Video pixel format.
 OH_AVPixelFormat pixelFormat = AV_PIXEL_FORMAT_NV12;
// Video width stride.
int32_t widthStride = 0;
// Video height stride.
int32_t heightStride = 0;
// Pointer to the encoder instance.
OH_AVCodec *videoEnc = nullptr;
// Encoder synchronization lock.
std::shared_mutex codecMutex;
// Encoder input queue.
CodecBufferQueue inQueue;
// Encoder output queue.
CodecBufferQueue outQueue;

[h2]Surface Mode

The following walks you through how to implement the entire video encoding process in surface mode and implement data rotation in asynchronous mode. In this example, surface data is input and encoded into a H.264 stream.

  1. Add the header files.

    #include <multimedia/player_framework/native_avcodec_videoencoder.h>
#include <multimedia/player_framework/native_avcapability.h>
#include <multimedia/player_framework/native_avcodec_base.h>
#include <multimedia/player_framework/native_avformat.h>
#include <multimedia/player_framework/native_avbuffer.h>
#include <fstream>

  2. Create an encoder instance.

    You can create an encoder by name or MIME type. In the code snippet below, the following variables are used:

    • videoEnc: pointer to the video encoder instance.

    • capability: pointer to the encoder's capability.

    • OH_AVCODEC_MIMETYPE_VIDEO_AVC: AVC video codec.

    The following is an example:

    // Create an encoder by name. If your application has special requirements, for example, expecting an encoder that supports a certain resolution, you can call OH_AVCodec_GetCapability to query the capability first.
OH_AVCapability *capability = OH_AVCodec_GetCapability(OH_AVCODEC_MIMETYPE_VIDEO_AVC, true);
// Create a hardware encoder instance.
OH_AVCapability *capability= OH_AVCodec_GetCapabilityByCategory(OH_AVCODEC_MIMETYPE_VIDEO_AVC, true, HARDWARE);
const char *codecName = OH_AVCapability_GetName(capability);
OH_AVCodec *videoEnc = OH_VideoEncoder_CreateByName(codecName);
    // Create an encoder by MIME type. Only specific codecs recommended by the system can be created in this way.
// Only hardware encoders can be created.
OH_AVCodec *videoEnc = OH_VideoEncoder_CreateByMime(OH_AVCODEC_MIMETYPE_VIDEO_AVC);

  3. Call OH_VideoEncoder_RegisterCallback() to register the callback functions.

    Register the OH_AVCodecCallback struct that defines the following callback function pointers:

    • OH_AVCodecOnError, a callback used to report a codec operation error. For details about the error codes, see OH_AVCodecOnError.

    • OH_AVCodecOnStreamChanged, a callback used to report a codec stream change, for example, format change.

    • OH_AVCodecOnNeedInputBuffer, a callback used to report input data required. This callback does not take effect, since you input data through the obtained surface.

    • OH_AVCodecOnNewOutputBuffer, a callback used to report output data generated, which means that encoding is complete.

    For details about the implementation of the callback function, see Sample Project.

    The following is an example:

    // Set the OH_AVCodecOnError callback function, which is used to report a codec operation error.
static void OnError(OH_AVCodec *codec, int32_t errorCode, void *userData)
{
    // Process the error code in the callback.
    (void)codec;
    (void)errorCode;
    (void)userData;
}
    // Set the OH_AVCodecOnStreamChanged callback function, which is used to report an encoding stream change.
static void OnStreamChanged(OH_AVCodec *codec, OH_AVFormat *format, void *userData)
{
    (void)codec;
    (void)format;
    (void)userData;
    // Obtain the video width and height after the resolution change through format.
    OH_AVFormat_GetIntValue(format, OH_MD_KEY_VIDEO_PIC_WIDTH, &width);
    OH_AVFormat_GetIntValue(format, OH_MD_KEY_VIDEO_PIC_HEIGHT, &height);
}
    // Set the OH_AVCodecOnNeedInputBuffer callback function, which is used to send an input frame to the data queue.
static void OnNeedInputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    // In surface mode, this callback function does not take effect. Data is input through the obtained surface.
    (void)userData;
    (void)index;
    (void)buffer;
}
    // Set the OH_AVCodecOnNewOutputBuffer callback function, which is used to send an encoded frame to the output queue.
static void OnNewOutputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    // The data buffer of the finished frame and its index are sent to outQueue.
    (void)codec;
    (void)userData;
    outQueue.Enqueue(std::make_shared<CodecBufferInfo>(index, buffer));
}
    // Call OH_VideoEncoder_RegisterCallback() to register the callback functions.
OH_AVCodecCallback cb = {&OnError, &OnStreamChanged, &OnNeedInputBuffer, &OnNewOutputBuffer};
OH_AVErrCode ret = OH_VideoEncoder_RegisterCallback(videoEnc, cb, nullptr); // nullptr: userData on which the callback depends is empty.
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

    In the callback functions, pay attention to multi-thread synchronization for operations on the data queue.

  4. (Optional) Call OH_VideoEncoder_RegisterParameterCallback() to register the frame-specific parameter callback function.

    For details, see Temporally Scalable Video Coding.

     // 5.1 Implement the OH_VideoEncoder_OnNeedInputParameter callback function.
 static void OnNeedInputParameter(OH_AVCodec *codec, uint32_t index, OH_AVFormat *parameter, void *userData)
 {
     // The index of the input frame parameter is sent to InParameterIndexQueue.
     // The input frame data (specified by parameter) is sent to InParameterQueue.
     // Data processing
     // Write the frame-specific parameter.
     // The value of OH_MD_KEY_VIDEO_ENCODER_QP_MAX must be greater than or equal to that of OH_MD_KEY_VIDEO_ENCODER_QP_MIN.
     OH_AVFormat_SetIntValue(parameter, OH_MD_KEY_VIDEO_ENCODER_QP_MAX, 30);
     OH_AVFormat_SetIntValue(parameter, OH_MD_KEY_VIDEO_ENCODER_QP_MIN, 20);
 }

 // 5.2 Register the frame-specific parameter callback function.
 OH_VideoEncoder_OnNeedInputParameter inParaCb = OnNeedInputParameter;
 OH_VideoEncoder_RegisterParameterCallback(videoEnc, inParaCb, NULL); // NULL: userData is null.

  5. Call OH_VideoEncoder_Configure() to configure the encoder.

    For details about the configurable options, see Video Dedicated Key-Value Pairs.

    For details about the parameter verification rules, see OH_VideoEncoder_Configure().

    The parameter value ranges can be obtained through the capability query interface. For details, see Obtaining Supported Codecs.

    Currently, the following options must be configured for all supported formats: video frame width, video frame height, and video pixel format.

    // Configure the video frame rate.
double frameRate = 30.0;
// Configure the video YUV range flag.
int32_t rangeFlag = 0;
// Configure the video primary color.
int32_t primary = static_cast<int32_t>(OH_ColorPrimary::COLOR_PRIMARY_BT709);
// Configure the transfer characteristics.
int32_t transfer = static_cast<int32_t>(OH_TransferCharacteristic::TRANSFER_CHARACTERISTIC_BT709);
// Configure the maximum matrix coefficient.
int32_t matrix = static_cast<int32_t>(OH_MatrixCoefficient::MATRIX_COEFFICIENT_IDENTITY);
// Configure the encoding profile.
int32_t profile = static_cast<int32_t>(OH_AVCProfile::AVC_PROFILE_HIGH);
// Configure the encoding bit rate mode.
int32_t rateMode = static_cast<int32_t>(OH_BitrateMode::BITRATE_MODE_VBR);
// Configure the key frame interval, in milliseconds.
int32_t iFrameInterval = 1000;
// Configure the SQR factor.
int32_t sqrFactor = 30;
// Configure the maximum bit rate, in bit/s.
int64_t maxBitRate = 20000000;
// Configure the bit rate, in bit/s.
int64_t bitRate = 5000000;
// Set the encoding quality.
int64_t quality = 90;

auto format = std::shared_ptr<OH_AVFormat>(OH_AVFormat_Create(), OH_AVFormat_Destroy);
if (format == nullptr) {
    // Handle exceptions.
}
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_WIDTH, width); // Mandatory.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_HEIGHT, height); // Mandatory.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_PIXEL_FORMAT, pixelFormat); // Mandatory.

OH_AVFormat_SetDoubleValue(format.get(), OH_MD_KEY_FRAME_RATE, frameRate);
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_RANGE_FLAG, rangeFlag);
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_COLOR_PRIMARIES, primary);
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_TRANSFER_CHARACTERISTICS, transfer);
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_MATRIX_COEFFICIENTS, matrix);
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_I_FRAME_INTERVAL, iFrameInterval);
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_PROFILE, profile);
// Configure OH_MD_KEY_QUALITY only when OH_BitrateMode is set to BITRATE_MODE_CQ.
if (rateMode == static_cast<int32_t>(OH_BitrateMode::BITRATE_MODE_CQ)) {
    OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_QUALITY, quality);
} else if (rateMode == static_cast<int32_t>(OH_BitrateMode::BITRATE_MODE_SQR)) {
    // Configure OH_MD_KEY_MAX_BITRATE and OH_MD_KEY_SQR_FACTOR only when OH_BitrateMode is set to BITRATE_MODE_SQR.
    OH_AVFormat_SetLongValue(format.get(), OH_MD_KEY_MAX_BITRATE, maxBitRate);
    OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_SQR_FACTOR, sqrFactor);
} else if (rateMode == static_cast<int32_t>(OH_BitrateMode::BITRATE_MODE_CBR) ||
           rateMode == static_cast<int32_t>(OH_BitrateMode::BITRATE_MODE_VBR) ||
           rateMode == static_cast<int32_t>(OH_BitrateMode::BITRATE_MODE_CBR_HIGH_QUALITY)){
    OH_AVFormat_SetLongValue(format.get(), OH_MD_KEY_BITRATE, bitRate);
}
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_VIDEO_ENCODE_BITRATE_MODE, rateMode);
OH_AVErrCode ret = OH_VideoEncoder_Configure(videoEnc, format.get());
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

    If an optional parameter is incorrectly configured, the error code AV_ERR_INVALID_VAL is returned. However, OH_VideoEncoder_Configure() does not fail. Instead, its execution continues with the default value.

  6. Obtain a surface.

    Obtain the OHNativeWindow in surface mode. The surface must be obtained before OH_VideoEncoder_Prepare is called.

    // Obtain the surface used for data input.
OHNativeWindow *nativeWindow;
OH_AVErrCode ret = OH_VideoEncoder_GetSurface(videoEnc, &nativeWindow);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}
// Use the OHNativeWindow* variable to obtain the address of the data to be filled through the producer interface.

    For details about how to use the OHNativeWindow* variable-type, see OHNativeWindow.

  7. Call OH_VideoEncoder_Prepare().

    This API prepares internal resources for the encoder.

    OH_AVErrCode ret = OH_VideoEncoder_Prepare(videoEnc);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  8. Call OH_VideoEncoder_Start() to start the encoder.

    // Configure the paths of the input and output files.
std::string_view outputFilePath = "/*yourpath*.h264";
std::unique_ptr<std::ofstream> outputFile = std::make_unique<std::ofstream>();
if (outputFile != nullptr) {
    outputFile->open(outputFilePath.data(), std::ios::out | std::ios::binary | std::ios::ate);
}
// Start the encoder.
OH_AVErrCode ret = OH_VideoEncoder_Start(videoEnc);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  9. (Optional) Call OH_VideoEncoder_SetParameter() to dynamically configure encoder parameters during running.

    OH_AVFormat *format = OH_AVFormat_Create();

// Dynamically request IDR frames.
OH_AVFormat_SetIntValue(format, OH_MD_KEY_REQUEST_I_FRAME, true);
// Dynamically reset the bit rate.
int64_t bitRate = 2000000;
OH_AVFormat_SetLongValue(format, OH_MD_KEY_BITRATE, bitRate);
// Dynamically reset the video frame rate.
double frameRate = 60.0;
OH_AVFormat_SetDoubleValue(format, OH_MD_KEY_FRAME_RATE, frameRate);
// The QP can be dynamically set.
// The value of OH_MD_KEY_VIDEO_ENCODER_QP_MAX must be greater than or equal to that of OH_MD_KEY_VIDEO_ENCODER_QP_MIN.
OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_QP_MAX, 30);
OH_AVFormat_SetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_QP_MIN, 20);

int32_t ret = OH_VideoEncoder_SetParameter(videoEnc, format);
if (ret != AV_ERR_OK) {
    // Exception handling.
}
OH_AVFormat_Destroy(format);

  10. Write the image to encode.

    In step 6, you have configured the OHNativeWindow* variable type returned by OH_VideoEncoder_GetSurface. The data required for encoding is continuously input by the surface. Therefore, you do not need to process the OnNeedInputBuffer callback function or use OH_VideoEncoder_PushInputBuffer to input data.

    This rule also applies in variable resolution scenarios.

  11. (Optional) Call OH_VideoEncoder_PushInputParameter() to notify the encoder that the frame-specific parameter configuration is complete.

    In step 4, you have registered the frame-specific parameter callback function.

    In the code snippet below, the following variables are used:

    • index: parameter passed by the callback function OnNeedInputParameter, which uniquely corresponds to the buffer.
    std::shared_ptr<CodecBufferInfo> bufferInfo = inQueue.Dequeue();
std::shared_lock<std::shared_mutex> lock(codecMutex);
if (bufferInfo == nullptr || !bufferInfo->isValid) {
    // Handle exceptions.
}
// You can determine the value.
int32_t isIFrame;
OH_AVFormat_SetIntValue(bufferInfo->parameter, OH_MD_KEY_REQUEST_I_FRAME, isIFrame);
OH_AVErrCode ret = OH_VideoEncoder_PushInputParameter(videoEnc, bufferInfo->index);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  12. Call OH_VideoEncoder_NotifyEndOfStream() to notify the encoder of EOS.

    // In surface mode, you only need to call this API to notify the encoder of EOS.
// In buffer mode, you need to set the AVCODEC_BUFFER_FLAGS_EOS flag and then call OH_VideoEncoder_PushInputBuffer to notify the encoder of EOS.
OH_AVErrCode ret = OH_VideoEncoder_NotifyEndOfStream(videoEnc);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  13. Call OH_VideoEncoder_FreeOutputBuffer() to release encoded frames.

    In the following example, the member variables of bufferInfo are as follows:

    • index: parameter passed by the callback function OnNewOutputBuffer, which uniquely corresponds to the buffer.

    • buffer: parameter passed by the callback function OnNewOutputBuffer. You can obtain the pointer to the shared memory address by calling OH_AVBuffer_GetAddr.

    • isValid: whether the buffer instance stored in bufferInfo is valid.

    std::shared_ptr<CodecBufferInfo> bufferInfo = outQueue.Dequeue();
std::shared_lock<std::shared_mutex> lock(codecMutex);
if (bufferInfo == nullptr || !bufferInfo->isValid) {
    // Handle exceptions.
}
int32_t qpAverage = 20;
double mseValue = 0.0;
OH_AVFormat *format = OH_AVBuffer_GetParameter(bufferInfo->buffer);
OH_AVFormat_GetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_QP_AVERAGE, &qpAverage);
OH_AVFormat_GetDoubleValue(format, OH_MD_KEY_VIDEO_ENCODER_MSE, &mseValue);
OH_AVFormat_Destroy(format);
// Obtain the encoded information.
OH_AVCodecBufferAttr info;
int32_t ret = OH_AVBuffer_GetBufferAttr(bufferInfo->buffer, &info);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}
// Write the encoded frame data (specified by buffer) to the output file.
outputFile->write(reinterpret_cast<char *>(OH_AVBuffer_GetAddr(bufferInfo->buffer)), info.size);
// Free the output buffer. index is the index of the buffer.
ret = OH_VideoEncoder_FreeOutputBuffer(videoEnc, bufferInfo->index);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  14. (Optional) Call OH_VideoEncoder_Flush() to refresh the encoder.

    After OH_VideoEncoder_Flush is called, the encoder remains in the Running state, but the input and output data and parameter set (such as the H.264 PPS/SPS) buffered in the encoder are cleared.

    To continue encoding, you must call OH_VideoEncoder_Start again.

    std::unique_lock<std::shared_mutex> lock(codecMutex);
// Refresh the encoder.
OH_AVErrCode flushRet = OH_VideoEncoder_Flush(videoEnc);
if (flushRet != AV_ERR_OK) {
    // Handle exceptions.
}
inQueue.Flush();
outQueue.Flush();
// Start encoding again.
OH_AVErrCode startRet = OH_VideoEncoder_Start(videoEnc);
if (startRet != AV_ERR_OK) {
    // Handle exceptions.
}

  15. (Optional) Call OH_VideoEncoder_Reset() to reset the encoder.

    After OH_VideoEncoder_Reset is called, the encoder returns to the initialized state. To continue encoding, you must call OH_VideoEncoder_Configure and then OH_VideoEncoder_Prepare.

    std::unique_lock<std::shared_mutex> lock(codecMutex);
// Reset the encoder.
OH_AVErrCode resetRet = OH_VideoEncoder_Reset(videoEnc);
if (resetRet != AV_ERR_OK) {
    // Handle exceptions.
}
inQueue.Flush();
outQueue.Flush();
// Reconfigure the encoder.
auto format = std::shared_ptr<OH_AVFormat>(OH_AVFormat_Create(), OH_AVFormat_Destroy);
if (format == nullptr) {
    // Handle exceptions.
}
OH_AVErrCode configRet = OH_VideoEncoder_Configure(videoEnc, format.get());
if (configRet != AV_ERR_OK) {
    // Handle exceptions.
}
// The encoder is ready again.
OH_AVErrCode prepareRet = OH_VideoEncoder_Prepare(videoEnc);
if (prepareRet != AV_ERR_OK) {
    // Handle exceptions.
}

  16. (Optional) Call OH_VideoEncoder_Stop() to stop the encoder.

    After OH_VideoEncoder_Stop is called, the encoder retains the encoding instance and releases the input and output buffers. You can call OH_VideoEncoder_Start to continue encoding.

    std::unique_lock<std::shared_mutex> lock(codecMutex);
// Stop the encoder.
OH_AVErrCode ret = OH_VideoEncoder_Stop(videoEnc);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}
inQueue.Flush();
outQueue.Flush();

  17. Call OH_VideoEncoder_Destroy() to destroy the encoder instance and release resources.

    1. This API cannot be called in the callback function.
    2. After the call, you must set a null pointer to the encoder to prevent program errors caused by wild pointers.
    std::unique_lock<std::shared_mutex> lock(codecMutex);
// Release the nativeWindow instance.
if(nativeWindow != nullptr){
    OH_NativeWindow_DestroyNativeWindow(nativeWindow);
    nativeWindow = nullptr;
}
// Call OH_VideoEncoder_Destroy to destroy the encoder.
OH_AVErrCode ret = AV_ERR_OK;
if (videoEnc != nullptr) {
    OH_VideoEncoder_Destroy(videoEnc);
    videoEnc = nullptr;
}
inQueue.Flush();
outQueue.Flush();

[h2]Buffer Mode

The following walks you through how to implement the entire video encoding process in buffer mode and implement data rotation in asynchronous mode. It uses the YUV file input and H.264 encoding format as an example.

  1. Add the header files.

    #include <multimedia/player_framework/native_avcodec_videoencoder.h>
#include <multimedia/player_framework/native_avcapability.h>
#include <multimedia/player_framework/native_avcodec_base.h>
#include <multimedia/player_framework/native_avformat.h>
#include <multimedia/player_framework/native_avbuffer.h>
#include <fstream>

  2. Create an encoder instance.

    The procedure is the same as that in surface mode and is not described here.

    // Create an encoder by name. If your application has special requirements, for example, expecting an encoder that supports a certain resolution, you can call OH_AVCodec_GetCapability to query the capability first.
OH_AVCapability *capability = OH_AVCodec_GetCapability(OH_AVCODEC_MIMETYPE_VIDEO_AVC, true);
const char *codecName = OH_AVCapability_GetName(capability);
OH_AVCodec *videoEnc = OH_VideoEncoder_CreateByName(codecName);
    // Create an encoder by MIME type. Only specific codecs recommended by the system can be created in this way.
// If multiple codecs need to be created, create hardware encoder instances first. If the hardware resources are insufficient, create software encoder instances.
OH_AVCodec *videoEnc = OH_VideoEncoder_CreateByMime(OH_AVCODEC_MIMETYPE_VIDEO_AVC);

  3. Call OH_VideoEncoder_RegisterCallback() to register the callback functions.

    Register the OH_AVCodecCallback struct that defines the following callback function pointers:

    • OH_AVCodecOnError, a callback used to report a codec operation error. For details about the error codes, see OH_AVCodecOnError.

    • OH_AVCodecOnStreamChanged, a callback used to report a codec stream change, for example, format change.

    • OH_AVCodecOnNeedInputBuffer, a callback used to report input data required, which means that the encoder is ready for receiving YUV/RGB data.

    • OH_AVCodecOnNewOutputBuffer, a callback used to report output data generated, which means that encoding is complete.

    You need to process the callback functions to ensure that the encoder runs properly.

    For details about the implementation of the callback function, see Sample Project.

    bool isFirstFrame = true;
int32_t qpAverage = 20;
double mseValue = 0.0;
    // Implement the OH_AVCodecOnError callback function.
static void OnError(OH_AVCodec *codec, int32_t errorCode, void *userData)
{
    // Process the error code in the callback.
    (void)codec;
    (void)errorCode;
    (void)userData;
}
    // Implement the OH_AVCodecOnStreamChanged callback function.
static void OnStreamChanged(OH_AVCodec *codec, OH_AVFormat *format, void *userData)
{
    // In buffer mode, this callback function does not take effect.
    (void)codec;
    (void)format;
    (void)userData;
}
    // Implement the OH_AVCodecOnNeedInputBuffer callback function.
static void OnNeedInputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    // Obtain the video width stride and height stride.
    if (isFirstFrame) {
        auto format = std::shared_ptr<OH_AVFormat>(OH_VideoEncoder_GetInputDescription(codec), OH_AVFormat_Destroy);
        if (format == nullptr) {
            // Handle exceptions.
        }
        bool ret = OH_AVFormat_GetIntValue(format.get(), OH_MD_KEY_VIDEO_STRIDE, &widthStride) &&
                   OH_AVFormat_GetIntValue(format.get(), OH_MD_KEY_VIDEO_SLICE_HEIGHT, &heightStride);
        if (!ret) {
            // Handle exceptions.
        }
        isFirstFrame = false;
    }
    // The data buffer of the input frame and its index are sent to inQueue.
    (void)codec;
    (void)userData;
    inQueue.Enqueue(std::make_shared<CodecBufferInfo>(index, buffer));
}
    // Implement the OH_AVCodecOnNewOutputBuffer callback function.
static void OnNewOutputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData)
{
    // Obtain the average QP and MSE of the video frame.
    OH_AVFormat *format = OH_AVBuffer_GetParameter(buffer);
    OH_AVFormat_GetIntValue(format, OH_MD_KEY_VIDEO_ENCODER_QP_AVERAGE, &qpAverage);
    OH_AVFormat_GetDoubleValue(format, OH_MD_KEY_VIDEO_ENCODER_MSE, &mseValue);
    OH_AVFormat_Destroy(format);
    // The data buffer of the finished frame and its index are sent to outQueue.
    (void)userData;
    outQueue.Enqueue(std::make_shared<CodecBufferInfo>(index, buffer));
}
    // Call OH_VideoEncoder_RegisterCallback() to register the callback functions.
OH_AVCodecCallback cb = {&OnError, &OnStreamChanged, &OnNeedInputBuffer, &OnNewOutputBuffer};
OH_AVErrCode ret = OH_VideoEncoder_RegisterCallback(videoEnc, cb, nullptr);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

    In the callback functions, pay attention to multi-thread synchronization for operations on the data queue.

  4. Call OH_VideoEncoder_Configure() to configure the encoder.

    The procedure is the same as that in surface mode and is not described here.

    auto format = std::shared_ptr<OH_AVFormat>(OH_AVFormat_Create(), OH_AVFormat_Destroy);
if (format == nullptr) {
    // Handle exceptions.
}
// Set the format.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_WIDTH, width); // Mandatory.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_HEIGHT, height); // Mandatory.
OH_AVFormat_SetIntValue(format.get(), OH_MD_KEY_PIXEL_FORMAT, pixelFormat); // Mandatory.
// Configure the encoder.
OH_AVErrCode ret = OH_VideoEncoder_Configure(videoEnc, format.get());
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  5. Call OH_VideoEncoder_Prepare().

    This API prepares internal resources for the encoder.

    OH_AVErrCode ret = OH_VideoEncoder_Prepare(videoEnc);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  6. Call OH_VideoEncoder_Start() to start the encoder.

    As soon as the encoder starts, the callback functions will be triggered to respond to events. Therefore, you must configure the input file and output file first.

    // Configure the paths of the input and output files.
std::string_view inputFilePath = "/*yourpath*.yuv";
std::string_view outputFilePath = "/*yourpath*.h264";
std::unique_ptr<std::ifstream> inputFile = std::make_unique<std::ifstream>();
std::unique_ptr<std::ofstream> outputFile = std::make_unique<std::ofstream>();
if (inputFile != nullptr) {
    inputFile->open(inputFilePath.data(), std::ios::in | std::ios::binary);
}
if (outputFile != nullptr) {
    outputFile->open(outputFilePath.data(), std::ios::out | std::ios::binary | std::ios::ate);
}
// Start the encoder.
OH_AVErrCode ret = OH_VideoEncoder_Start(videoEnc);
if (ret != AV_ERR_OK) {
    // Handle exceptions.
}

  7. (Optional) Dynamically configure encoder parameters during running.

    OH_AVFormat *format = OH_AVFormat_Create();

// Dynamically request IDR frames.
OH_AVFormat_SetIntValue(format, OH_MD_KEY_REQUEST_I_FRAME, true);
// Dynamically reset the bit rate.
int64_t bitRate = 2000000;
OH_AVFormat_SetLongValue(format, OH_MD_KEY_BITRATE, bitRate);
// Dynamically reset the video frame rate.
double frameRate = 60.0;
OH_AVFormat_SetDoubleValue(format, OH_MD_KEY_FRAME_RATE, frameRate);

int32_t ret = OH_VideoEncoder_SetParameter(videoEnc, format);
if (ret != AV_ERR_OK) {
    // Exception handling.
}
OH_AVFormat_Destroy(format);

  8. Call OH_VideoEncoder_PushInputBuffer() to push the image to the input queue for encoding.

    In the code snippet below, the following variables are used:

    • widthStride: width stride of the obtained buffer data.

    • heightStride: height stride of the obtained buffer data.

    The member variables of bufferInfo are as follows:

    • buffer: parameter passed by the callback function OnNeedInputBuffer. You can obtain the pointer to the shared memory address by calling OH_AVBuffer_GetAddr.

    • index: parameter passed by the callback function OnNeedInputBuffer, which uniquely corresponds to the buffer.

    • isValid: whether the buffer instance stored in bufferInfo is valid.

    std::shared_ptr<CodecBufferInfo> bufferInfo = inQueue.Dequeue();
std::shared_lock<std::shared_mutex> lock(codecMutex);
if (bufferInfo == nullptr || !bufferInfo->isValid) {
    // Handle exceptions.
}
// Write image data.
int32_t frameSize = 0;
if (widthStride == width && heightStride == height) {
    frameSize = width * height * 3 / 2; // Formula for calculating the data size of each frame in NV12 pixel format.
    int32_t capacity = OH_AVBuffer_GetCapacity(bufferInfo->buffer);
    if (frameSize > capacity) {
        // Handle exceptions.
    }
    // Process the file stream and obtain the frame length, and then write the data to encode to the buffer of the specified index.
    uint8_t *addr = OH_AVBuffer_GetAddr(bufferInfo->buffer);
    if (addr == nullptr) {
       // Handle exceptions.
    }
    if (inputFile != nullptr && inputFile->is_open()) {
        inputFile->read(reinterpret_cast<char *>(addr), frameSize);
    }
} else {
    // If the stride is not equal to the width, perform offset based on the stride. For details, see the following example.
}
// Configure the buffer information.
OH_AVCodecBufferAttr info;
info.size = frameSize;
info.offset = 0;
// Unlike the surface mode, in buffer mode, the application must explicitly set pts. Compute it based on the intended display time, for example, frameIndex * 1000000 / frameRate.
info.pts = 0;
// To prevent the flags from being randomly initialized to AVCODEC_BUFFER_FLAGS_EOS and causing abnormal usage, the flags must be assigned a value such as 0 (identifier for normal frames).
info.flags = 0;
OH_AVErrCode setBufferRet = OH_AVBuffer_SetBufferAttr(bufferInfo->buffer, &info);
if (setBufferRet != AV_ERR_OK) {
    // Handle exceptions.
}
// Configure the buffer frame-specific information.
// You can determine the value.
int32_t isIFrame;
auto parameter = std::shared_ptr<OH_AVFormat>(OH_AVBuffer_GetParameter(bufferInfo->buffer), OH_AVFormat_Destroy);
if (parameter == nullptr) {
    // Handle exceptions.
}
OH_AVFormat_SetIntValue(parameter.get(), OH_MD_KEY_REQUEST_I_FRAME, isIFrame);
OH_AVErrCode parameterRet = OH_AVBuffer_SetParameter(bufferInfo->buffer, parameter.get());
if (parameterRet != AV_ERR_OK) {
    // Handle exceptions.
}
// Send the data to the input buffer for encoding. index is the index of the buffer.
OH_AVErrCode pushInputRet = OH_VideoEncoder_PushInputBuffer(videoEnc, bufferInfo->index);
if (pushInputRet != AV_ERR_OK) {
    // Handle exceptions.
}

    Offset the stride. The following uses an NV12 image as an example,

    presenting the image layout of width, height, wStride, and hStride.

    • OH_MD_KEY_WIDTH corresponds to width.

    • OH_MD_KEY_HEIGHT corresponds to height.

    • OH_MD_KEY_VIDEO_STRIDE corresponds to wStride.

    • OH_MD_KEY_VIDEO_SLICE_HEIGHT corresponds to hStride.

    Figure 3 NV12 image layout

    Add the header files.

    #include <string.h>

    The sample code is as follows:

    struct Rect // Width and height of the source buffer. You can set them as required.
{
    int32_t width;
    int32_t height;
};

struct DstRect // Width stride and height stride of the destination buffer. They are obtained by calling OH_VideoEncoder_GetInputDescription.
{
    int32_t wStride;
    int32_t hStride;
};

struct SrcRect // Width stride and height stride of the source buffer. You can set them as required.
{
    int32_t wStride;
    int32_t hStride;
};

Rect rect = {320, 240};
DstRect dstRect = {320, 256};
SrcRect srcRect = {320, 250};
uint8_t* dst = new uint8_t[dstRect.hStride * dstRect.wStride * 3 / 2]; // Pointer to the target memory area.
uint8_t* src = new uint8_t[srcRect.hStride * srcRect.wStride * 3 / 2]; // Pointer to the source memory area.
uint8_t* dstTemp = dst;
uint8_t* srcTemp = src;
rect.height = ((rect.height + 1) / 2)  * 2; // This ensures the height is always even.
rect.width = ((rect.width + 1) / 2)  * 2; // This ensures the width is always even.

// Y: Copy the source data in the Y region to the target data in another region.
for (int32_t i = 0; i < rect.height; ++i) {
    // Copy a row of data from the source to a row of the target.
    memcpy(dstTemp, srcTemp, rect.width);
    // Update the pointers to the source data and target data to copy the next row. The pointers to the source data and target data are moved downwards by one wStride each time the source data and target data are updated.
    dstTemp += dstRect.wStride;
    srcTemp += srcRect.wStride;
}
// Padding.
// Update the pointers to the source data and target data. The pointers move downwards by one padding.
dstTemp += (dstRect.hStride - rect.height) * dstRect.wStride;
srcTemp += (srcRect.hStride - rect.height) * srcRect.wStride;
rect.height >>= 1;
// UV: Copy the source data in the UV region to the target data in another region.
for (int32_t i = 0; i < rect.height; ++i) {
    memcpy(dstTemp, srcTemp, rect.width);
    dstTemp += dstRect.wStride;
    srcTemp += srcRect.wStride;
}

delete[] dst;
dst = nullptr;
delete[] src;
src = nullptr;

    When processing buffer data (before pushing data) during hardware encoding, you must copy the image data after width and height alignment to the input callback AVBuffer.

    Generally, copy the image width, height, stride, and pixel format to ensure correct processing of the data to encode.

    For details, see step 3 in Buffer Mode.

    The following figures show the layout of common image formats.

    Figure 4 YUVI420

    Figure 5 RGBA1010102

  9. Notify the encoder of EOS.

    During encoding, once the last frame of data is sent to the input buffer for encoding, set the flag of bufferInfo to AVCODEC_BUFFER_FLAGS_EOS to notify the encoder that the input is complete.

    In the following example, the member variables of bufferInfo are as follows:

    • index: parameter passed by the callback function OnNeedInputBuffer, which uniquely corresponds to the buffer.

    • buffer: parameter passed by the callback function OnNeedInputBuffer. You can obtain the pointer to the shared memory address by calling OH_AVBuffer_GetAddr.

    • isValid: whether the buffer instance stored in bufferInfo is valid.

    std::shared_ptr<CodecBufferInfo> bufferInfo = inQueue.Dequeue();
std::shared_lock<std::shared_mutex> lock(codecMutex);
if (bufferInfo == nullptr || !bufferInfo->isValid) {
    // Handle exceptions.
}
// Write the last frame of image data. For details, see step 8.
// Configure the buffer information and set the AVCODEC_BUFFER_FLAGS_EOS flag.
OH_AVCodecBufferAttr info;
info.size = frameSize;
info.offset = 0;
// Unlike the surface mode, in buffer mode, the application must explicitly set pts. Compute it based on the intended display time, for example, frameIndex * 1000000 / frameRate.
info.pts = 0;
info.flags = AVCODEC_BUFFER_FLAGS_EOS;
OH_AVErrCode setBufferRet = OH_AVBuffer_SetBufferAttr(bufferInfo->buffer, &info);
if (setBufferRet != AV_ERR_OK) {
    // Handle exceptions.
}
OH_AVErrCode pushInputRet = OH_VideoEncoder_PushInputBuffer(videoEnc, bufferInfo->index);
if (pushInputRet != AV_ERR_OK) {
    // Handle exceptions.
}

  10. Call OH_VideoEncoder_FreeOutputBuffer() to release encoded frames.

    The procedure is the same as that in surface mode and is not described here.

    std::shared_ptr<CodecBufferInfo> bufferInfo = outQueue.Dequeue();
std::shared_lock<std::shared_mutex> lock(codecMutex);
if (bufferInfo == nullptr || !bufferInfo->isValid) {
    // Handle exceptions.
}
// Obtain the encoded information.
OH_AVCodecBufferAttr info;
OH_AVErrCode getBufferRet = OH_AVBuffer_GetBufferAttr(bufferInfo->buffer, &info);
if (getBufferRet != AV_ERR_OK) {
    // Handle exceptions.
}
// Write the encoded frame data (specified by buffer) to the output file.
uint8_t *addr = OH_AVBuffer_GetAddr(bufferInfo->buffer);
if (addr == nullptr) {
   // Handle exceptions.
}
if (outputFile != nullptr && outputFile->is_open()) {
    outputFile->write(reinterpret_cast<char *>(addr), info.size);
}
// Free the output buffer. index is the index of the buffer.
OH_AVErrCode freeOutputRet = OH_VideoEncoder_FreeOutputBuffer(videoEnc, bufferInfo->index);
if (freeOutputRet != AV_ERR_OK) {
    // Handle exceptions.
}

The subsequent processes (including refreshing, resetting, stopping, and destroying the encoder) are the same as those in surface mode. For details, see steps 14-17 in Surface Mode.

Notes

  1. Buffer mode does not support 10-bit image data.

  2. Due to limited hardware encoder resources, you must call OH_VideoEncoder_Destroy to destroy every encoder instance and release resources when the encoder is no longer needed.

  3. The Flush, Reset, Stop, and Destroy APIs must be called in a non-callback thread. When these APIs are executed, they block and wait until all triggered callbacks are complete, and then return the execution result.

  4. Once Flush, Reset, or Stop is called, the system reclaims OH_AVBuffer. Do not continue to operate the OH_AVBuffer obtained through the previous callback function.

  5. In buffer mode, after obtaining the pointer to an OH_AVBuffer instance through the input callback function OH_AVCodecOnNeedInputBuffer, you must call OH_VideoEncoder_PushInputBuffer to notify the system that the instance has been used. This ensures that the system can encode the data in the instance. If you obtain the OH_NativeBuffer pointer instance by calling OH_AVBuffer_GetNativeBuffer and the lifecycle of this instance exceeds that of the current OH_AVBuffer pointer instance, you need to manually copy the data and manage the lifecycle of the newly generated OH_NativeBuffer instance to ensure that it is correctly used and released.

Video Encoding Capabilities

Capability How to Use
Configuring the encoder parameters, including the frame rate, bit rate, and QPMin/QPMax Call OH_VideoEncoder_SetParameter(). For details, see step 9 in Surface Input below.
Setting the encoding QPMin/QPMax on a per-frame basis Call OH_VideoEncoder_RegisterParameterCallback(). For details, see step 4 in Surface Input below.
Layered encoding and LTR setting For details, see Temporally Scalable Video Coding.
Obtaining the average QP and MSE of each frame Call OnNewOutputBuffer(). For details, see step 3 in Surface Input below.
Variable resolution The encoder supports the change of the input image resolution. Currently, this is supported only in surface mode, and the image width and height cannot exceed those configured by calling OH_VideoEncoder_Configure. For details, see step 5 in Surface Input below.