目录

Rendering Video Frames

Overview

Rendering video frames is the process of converting raw video data (such as YUV and RGB formats) into displayable images on the screen, and outputting to display devices like smartphones. This process is key in scenarios such as video playback, live streaming, and camera previews, affecting visual quality.

This guide uses video decoding as an example to explain how to render decoded video data onto the device screen. Currently, the system offers various methods for rendering video frames, including using the XComponent, Open Graphics Library (OpenGL), and Vulkan. The advantages, disadvantages, and usage scenarios for these three solutions are outlined in the table below.

Solution Advantage Disadvantage Scenario
Rendering with XComponent

Low development barrier; directly interfaces with the decoder via NativeWindow of the XComponent.

No extra processing for decoded YUV video data, which can be directly rendered by the XComponent.

Limited customization; insufficient for complex scenarios.

Minimal performance optimization potential; limited performance in demanding scenarios due to its non-customizable rendering pipeline.

 Suitable for simple video playback without complex frame processing.
Rendering with OpenGL

Supports complex scenarios with customizable development.

Offers cross-language, cross-platform APIs for application development and migration.

Moderate development barrier; rich ecosystem.

 Requires additional processing of video data formats (usually conversion to RGBA format). Ideal for complex scenarios like live streaming with bullet comments.
Rendering with Vulkan

Provides fine-grained control over the rendering pipeline for optimized performance.

Can handle complex scenarios.

Offers cross-language, cross-platform APIs for application development and migration.

Requires additional processing of video data formats (usually conversion to RGBA format).

Higher development barrier; requires manual memory and command buffer management, increasing development complexity and debugging difficulty.

 Ideal for complex scenarios like live streaming with bullet comments.

Rendering with XComponent

[h2]Scenario

The system provides the AVPlayer and Video components for playing videos. However, the supported file formats are limited. When videos in other formats (such as RMVB) or with digital copyright protection are played, your requirements cannot be met. In this case, you can use AVCodec to decode the video and render the decoded frames directly via XComponent. This approach is straightforward and commonly used, allowing you to pass decoded YUV data to XComponent without handling format conversions.

[h2]How to Implement

This solution requires obtaining the NativeWindow object associated with XComponent and setting the object to the video decoder during initiation. The specific implementation steps are as follows:

  1. During the creation of XComponent, obtain its NativeWindow object via the OnSurfaceCreatedCB callback.

  2. Initialize the video decoding environment, including initializing the demuxer and decoder.

  3. Start the decoder, decoder input thread, and decoder output thread.

  4. Obtain an available AVBuffer through OnNeedInputBuffer, fill the AVBuffer with the video data read by the demuxer in the decoding input thread, and submit it to the decoder for decoding.

  5. Obtain the decoded video data through OnNeedOutputBuffer, and submit the data to the output surface (the NativeWindow object of XComponent) in the decoding output thread.

[h2]How to Develop

  1. During the creation of XComponent, obtain its NativeWindow object via the OnSurfaceCreatedCB callback.

    void OnSurfaceCreatedCB(OH_NativeXComponent *component, void *window)
{
    OH_LOG_Print(LOG_APP, LOG_INFO, LOG_PRINT_DOMAIN, "Callback", "OnSurfaceCreatedCB");
    if ((component == nullptr) || (window == nullptr)) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "Callback",
                     "onSurfaceCreatedCB: component or window is null");
        return;
    }
    char idStr[OH_XCOMPONENT_ID_LEN_MAX + 1] = {'\0'};
    uint64_t idSize = OH_XCOMPONENT_ID_LEN_MAX + 1;
    if (OH_NativeXComponent_GetXComponentId(component, idStr, &idSize) != OH_NATIVEXCOMPONENT_RESULT_SUCCESS) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "Callback",
                     "OnSurfaceCreatedCB:Unable to get XComponent id");
        return;
    }
    std::string id(idStr);
    auto render = PluginRender::GetInstance(id);
    uint64_t width;
    uint64_t height;
    int32_t xSize = OH_NativeXComponent_GetXComponentSize(component, window, &width, &height);
    if ((xSize != OH_NATIVEXCOMPONENT_RESULT_SUCCESS) || (render == nullptr)) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "Callback",
                     "OnSurfaceCreatedCB: Unable to qet XComponent size");
        return;
    }
    render->nativeWindow = reinterpret_cast<OHNativeWindow *>(window);
    (void)OH_NativeWindow_NativeWindowHandleOpt(render->nativeWindow, SET_BUFFER_GEOMETRY, static_cast<int>(width),
                                                    static_cast<int>(height));
    if (id == "OpenGL") {
        render->openGLRenderThread_->UpdateNativeWindow(render->nativeWindow, width, height);
    } else if(id == "Vulkan") {
        render->vulkanRenderThread_->UpdateNativeWindow(render->nativeWindow, width, height);
    }
}

  2. Initialize the video decoding environment.

    int32_t Player::Init(SampleInfo &sampleInfo) {
    std::unique_lock<std::mutex> lock(mutex_);
    if(isStarted_ || (demuxer_ != nullptr && videoDecoder_ != nullptr)) {
        OH_LOG_ERROR(LOG_APP, "Already started.");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    sampleInfo_ = sampleInfo;

    videoDecoder_ = std::make_unique<VideoDecoder>();
    demuxer_ = std::make_unique<Demuxer>();
    isReleased_ = false;
    int32_t ret = demuxer_->Create(sampleInfo_);
    if (ret != AVCODEC_SAMPLE_ERR_OK) {
        OH_LOG_ERROR(LOG_APP, "Create demuxer failed");
        doneCond_.notify_all();
        lock.unlock();
        StartRelease();
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    ret = CreateVideoDecoder();
    if (ret != AVCODEC_SAMPLE_ERR_OK) {
        OH_LOG_ERROR(LOG_APP, "Create video decoder failed");
        doneCond_.notify_all();
        lock.unlock();
        StartRelease();
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    OH_LOG_INFO(LOG_APP, "Succeed");
    return AVCODEC_SAMPLE_ERR_OK;
}

    Create a demuxer.

    int32_t Demuxer::Create(SampleInfo &info) {
    source_ = OH_AVSource_CreateWithFD(info.inputFd, info.inputFileOffset, info.inputFileSize);
    if (source_ == nullptr) {
        OH_LOG_ERROR(LOG_APP,
                     "Create demuxer source failed, fd: %{public}d, offset: %{public}ld, file size: %{public}ld",
                     info.inputFd, info.inputFileOffset, info.inputFileSize);
        return AVCODEC_SAMPLE_ERR_ERROR;
    }
    
    demuxer_ = OH_AVDemuxer_CreateWithSource(source_);
    if (demuxer_ == nullptr) {
        OH_LOG_ERROR(LOG_APP, "Create demuxer failed");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    auto sourceFormat = std::shared_ptr<OH_AVFormat>(OH_AVSource_GetSourceFormat(source_), OH_AVFormat_Destroy);
    if (sourceFormat == nullptr) {
        OH_LOG_ERROR(LOG_APP, "Get source format failed");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    int32_t ret = GetTrackInfo(sourceFormat, info);
    if (ret != AVCODEC_SAMPLE_ERR_OK) {
        OH_LOG_ERROR(LOG_APP, "Get video track info, ret: %{public}d", ret);
        return AVCODEC_SAMPLE_ERR_ERROR;
    }
    
    return AVCODEC_SAMPLE_ERR_OK;
}

    Create a video decoder.

    int32_t Player::CreateVideoDecoder() {
    OH_LOG_INFO(LOG_APP, "video mime:%{public}s", sampleInfo_.videoCodecMime.c_str());
    int32_t ret = videoDecoder_->Create(sampleInfo_.videoCodecMime);
    if (ret != AVCODEC_SAMPLE_ERR_OK) {
        OH_LOG_INFO(LOG_APP, "Create video decoder failed, mime:%{public}s", sampleInfo_.videoCodecMime.c_str());
    } else {
        videoDecContext_ = new CodecUserData;
        ret = videoDecoder_->Config(sampleInfo_, videoDecContext_);
        if(!(ret == AVCODEC_SAMPLE_ERR_OK)) {
            OH_LOG_ERROR(LOG_APP, "Video Decoder config failed");
            return ret;
        }
    }
    return AVCODEC_SAMPLE_ERR_OK;
}

    Set the surface for the video decoder.

    int32_t VideoDecoder::Config(const SampleInfo &sampleInfo, CodecUserData *codecUserData) {
    if (decoder_ == nullptr) {
        OH_LOG_ERROR(LOG_APP, "Decoder is null");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    if (codecUserData == nullptr) {
        OH_LOG_ERROR(LOG_APP, "Invalid param: codecUserData");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }
    
    int32_t ret = Configure(sampleInfo);
    if (ret != AVCODEC_SAMPLE_ERR_OK) {
        OH_LOG_ERROR(LOG_APP, "Configure failed");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    if (sampleInfo.window != nullptr) {
        int ret = OH_VideoDecoder_SetSurface(decoder_, sampleInfo.window);
        if (ret != AV_ERR_OK) {
            OH_LOG_ERROR(LOG_APP, "Set surface failed, ret: %{public}d", ret);
            return AVCODEC_SAMPLE_ERR_ERROR;
        }
    }
    
    ret = SetCallback(codecUserData);
    if (ret != AVCODEC_SAMPLE_ERR_OK) {
        OH_LOG_ERROR(LOG_APP, "Set callback failed, ret: %{public}d", ret);
        return AVCODEC_SAMPLE_ERR_ERROR;
    }
    
    ret = OH_VideoDecoder_Prepare(decoder_);
    if (ret != AV_ERR_OK) {
        OH_LOG_ERROR(LOG_APP, "Prepare failed, ret: %{public}d", ret);
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    return AVCODEC_SAMPLE_ERR_OK;
}

  3. Start the decoder, decoder input thread, and decoder output thread.

    int32_t Player::Start() {
    std::unique_lock<std::mutex> lock(mutex_);
    int32_t ret;
    if (isStarted_ || demuxer_ == nullptr) {
        OH_LOG_ERROR(LOG_APP, "Already started.");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }
    
    if (videoDecContext_) {
        ret = videoDecoder_->Start();
        if (ret != AVCODEC_SAMPLE_ERR_OK) {
            OH_LOG_ERROR(LOG_APP, "Video Decoder start failed");
            lock.unlock();
            StartRelease();
            return AVCODEC_SAMPLE_ERR_ERROR;
        }
        isStarted_ = true;
        videoDecInputThread_ = std::make_unique<std::thread>(&Player::VideoDecInputThread, this);
        videoDecOutputThread_ = std::make_unique<std::thread>(&Player::VideoDecOutputThread, this);

        if (videoDecInputThread_ == nullptr || videoDecOutputThread_ == nullptr) {
            OH_LOG_ERROR(LOG_APP, "Create thread failed");
            lock.unlock();
            StartRelease();
            return AVCODEC_SAMPLE_ERR_ERROR;
        }
    }

    OH_LOG_INFO(LOG_APP, "Succeed");
    doneCond_.notify_all();
    return AVCODEC_SAMPLE_ERR_OK;
}

  4. Obtain an available AVBuffer via the OnNeedInputBuffer callback, and add the AVBuffer to the queue for subsequent processing.

    void SampleCallback::OnNeedInputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData) {
    if (userData == nullptr) {
        return;
    }
    (void)codec;
    CodecUserData *codecUserData = static_cast<CodecUserData *>(userData);
    std::unique_lock<std::mutex> lock(codecUserData->inputMutex);
    codecUserData->inputBufferInfoQueue.emplace(index, buffer);
    codecUserData->inputCond.notify_all();
}

    Fill the AVBuffer with video data in the decoding input thread and queue it to the decoder for decoding.

    void Player::VideoDecInputThread() {
    while (true) {
        if (!isStarted_) {
            OH_LOG_ERROR(LOG_APP, "Decoder input thread out");
            break;;
        }
        
        std::unique_lock<std::mutex> lock(videoDecContext_->inputMutex);
        bool condRet = videoDecContext_->inputCond.wait_for(
            lock, 5s, [this]() { return !isStarted_ || !videoDecContext_->inputBufferInfoQueue.empty(); });
        if (!isStarted_) {
            OH_LOG_ERROR(LOG_APP, "Work done, thread out");
            break;
        }
        if (videoDecContext_->inputBufferInfoQueue.empty()) {
            OH_LOG_ERROR(LOG_APP, "Buffer queue is empty, continue, cond ret: %{public}d", condRet);
            continue;
        }

        CodecBufferInfo bufferInfo = videoDecContext_->inputBufferInfoQueue.front();
        videoDecContext_->inputBufferInfoQueue.pop();
        videoDecContext_->inputFrameCount++;
        lock.unlock();

        demuxer_->ReadSample(demuxer_->GetVideoTrackId(), reinterpret_cast<OH_AVBuffer *>(bufferInfo.buffer),
                             bufferInfo.attr);

        int32_t ret = videoDecoder_->PushInputBuffer(bufferInfo);
        if (ret != AVCODEC_SAMPLE_ERR_OK) {
            OH_LOG_ERROR(LOG_APP, "Push data failed, thread out");
            break;
        }

        if (bufferInfo.attr.flags & AVCODEC_BUFFER_FLAGS_EOS) {
            OH_LOG_ERROR(LOG_APP, "Catch EOS, thread out");
            break;
        }
    }
}

  5. In the decoding output thread, submit the decoded video data to the output surface. Obtain the decoded video data via the OnNeedOutputBuffer callback.

    void SampleCallback::OnNewOutputBuffer(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *buffer, void *userData) {
    if (userData == nullptr) {
        return;
    }
    (void)codec;
    CodecUserData *codecUserData = static_cast<CodecUserData *>(userData);
    std::unique_lock<std::mutex> lock(codecUserData->outputMutex);
    codecUserData->outputBufferInfoQueue.emplace(index, buffer);
    codecUserData->outputCond.notify_all();
}

    In the decoding output thread, submit the decoded video data to the output surface. The surface has been set to the NativeWindow object of the XComponent during decoder initialization.

    void Player::VideoDecOutputThread() {
    sampleInfo_.frameInterval = MICROSECOND / sampleInfo_.frameRate;
    while (true) {
        thread_local auto lastPushTime = std::chrono::system_clock::now();
        if (!isStarted_) {
            OH_LOG_ERROR(LOG_APP, "Decoder output thread out");
            break;
        }
        std::unique_lock<std::mutex> lock(videoDecContext_->outputMutex);
        bool condRet = videoDecContext_->outputCond.wait_for(
            lock, 5s, [this]() { return !isStarted_ || !videoDecContext_->outputBufferInfoQueue.empty(); });
        if (!isStarted_) {
            OH_LOG_ERROR(LOG_APP, "Decoder output thread out");
            break;
        }
        if (videoDecContext_->outputBufferInfoQueue.empty()) {
            OH_LOG_ERROR(LOG_APP, "Buffer queue is empty, continue, cond ret: %{public}d", condRet);
            continue;
        }

        CodecBufferInfo bufferInfo = videoDecContext_->outputBufferInfoQueue.front();
        videoDecContext_->outputBufferInfoQueue.pop();
        if (bufferInfo.attr.flags & AVCODEC_BUFFER_FLAGS_EOS) {
            OH_LOG_ERROR(LOG_APP, "Catch EOS, thread out");
            break;
        }
        videoDecContext_->outputFrameCount++;
        OH_LOG_INFO(LOG_APP,"Out buffer count: %{public}u, size: %{public}d, flag: %{public}u, pts: %{public}ld",
                            videoDecContext_->outputFrameCount, bufferInfo.attr.size, bufferInfo.attr.flags,
                            bufferInfo.attr.pts);
        lock.unlock();

        int32_t ret = videoDecoder_->FreeOutputBuffer(bufferInfo.bufferIndex, true);
        if (ret != AVCODEC_SAMPLE_ERR_OK) {
            OH_LOG_ERROR(LOG_APP, "Decoder output thread out");
            break;
        }

        std::this_thread::sleep_until(lastPushTime + std::chrono::microseconds(sampleInfo_.frameInterval));
        lastPushTime = std::chrono::system_clock::now();
    }
    StartRelease();
}

    Call OH_VideoDecoder_RenderOutputBuffer to submit the video data to the output surface.

    int32_t VideoDecoder::FreeOutputBuffer(uint32_t bufferIndex, bool render) {
    if (decoder_ == nullptr) {
        OH_LOG_ERROR(LOG_APP, "Decoder is null");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }

    int32_t ret = AVCODEC_SAMPLE_ERR_OK;
    if (render) {
        ret = OH_VideoDecoder_RenderOutputBuffer(decoder_, bufferIndex);
    } else {
        ret = OH_VideoDecoder_FreeOutputBuffer(decoder_, bufferIndex);
    }
    if (ret != AV_ERR_OK) {
        OH_LOG_ERROR(LOG_APP, "Free output data failed");
        return AVCODEC_SAMPLE_ERR_ERROR;
    }
    return AVCODEC_SAMPLE_ERR_OK;
}

Rendering with OpenGL

[h2]Scenario

Open Graphics Library (OpenGL) is a cross-language, cross-platform API for rendering 2D and 3D vector graphics. While the XComponent-based video rendering solution is straightforward and supports numerous video formats, it offers limited flexibility for video frame processing operations such as zooming and rotation. With OpenGL, you can implement customized image processing features like video rotation and bullet comments as required.

[h2]How to Implement

To render video frames using OpenGL, you need to use NativeImage to connect the decoder with OpenGL. You should first call OH_NativeImage_AcquireNativeWindow() to obtain the NativeWindow object associated with NativeImage and set this object to the decoder to consume the video frame data generated by the decoder. Then, call OH_NativeImage_UpdateSurfaceImage() to update the video frame buffer obtained by NativeImage into an OpenGL-related texture. Finally, call eglSwapBuffers to submit the rendered buffer to the NativeWindow object of XComponent.

The specific implementation steps are as follows:

  1. Start the rendering thread and initialize the OpenGL environment, including obtaining the display device, initializing the EGL environment, and creating the OpenGL context.

  2. Create a NativeImage object and obtain its NativeWindow object.

  3. Obtain the NativeWindow object of XComponent and create an EGLSurface based on this object.

  4. Initialize the video decoding environment, including the demuxer and decoder.

    Unlike the XComponent-based rendering solution, in the decoder configuration, the surface input parameter is the NativeWindow object of NativeImage, not the NativeWindow object of XComponent.

  5. Start the decoder, decoder input thread, and decoder output thread.

  6. Obtain an available AVBuffer through OnNeedInputBuffer, fill the AVBuffer with the video data read by the demuxer in the decoding input thread, and submit it to the decoder for decoding.

  7. Obtain the decoded video data through OnNeedOutputBuffer, and submit the data to the output surface (the NativeWindow object of NativeImage) in the decoding output thread.

  8. Update the video frame buffer to the OpenGL texture via NativeImage.

  9. Call eglSwapBuffers to swap the front and back buffers, and display the rendering result on the screen.

[h2]How to Develop

  1. Start the rendering thread and initialize the OpenGL environment.

    void OpenGLRenderThread::ThreadMainLoop()
{
    threadId_ = std::this_thread::get_id();
    if (!InitRenderContext()) {
        return;
    }
    if (!InitNativeVsync()) {
        return;
    }
    if (!CreateNativeImage()) {
        return;
    }
    while (running_) {
        {
            std::unique_lock<std::mutex> lock(wakeUpMutex_);
            wakeUpCond_.wait(lock, [this]() { return wakeUp_ || vSyncCnt_ > 0 || availableFrameCnt_ > 0; });
            wakeUp_ = false;
            vSyncCnt_--;
            (void)OH_NativeVSync_RequestFrame(nativeVsync_, &OpenGLRenderThread::OnVsync, this);
        }
        
        std::vector<RenderTask> tasks;
        {
            std::unique_lock<std::mutex> lock(taskMutex_);
            tasks.swap(tasks_);
        }
        for (const auto &task : tasks) {
            task(*renderContext_);
        }

        if (availableFrameCnt_ <= 0) {
            continue;
        }
        DrawImage();
        availableFrameCnt_--;
    }
}

    Initialize the OpenGL environment, including obtaining the display, initializing EGL, and creating an EGL context.

    bool OpenGLRender::Init()
{
    if (IsEglContextReady()) {
        return true;
    }

    OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "EglRenderContext", "EglRenderContext::Init begin.");
    eglDisplay_ = GetPlatformEglDisplay(EGL_PLATFORM_OHOS_KHR, EGL_DEFAULT_DISPLAY, NULL);
    if (eglDisplay_ == EGL_NO_DISPLAY) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "EglRenderContext",
                     "EglRenderContext::Init: failed to create eglDisplay, error: %{public}s.", GetEglErrorString());
        return false;
    }

    EGLint major;
    EGLint minor;
    if (eglInitialize(eglDisplay_, &major, &minor) == EGL_FALSE) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "EglRenderContext",
                     "EglRenderContext::Init: Failed to initialize EGLDisplay, error: %{public}s.",
                     GetEglErrorString());
    }
    SetupEglExtensions();

    if (eglBindAPI(EGL_OPENGL_ES_API) == EGL_FALSE) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "EglRenderContext",
                     "EglRenderContext::Init: Failed to bind OpenGL ES API, error: %{public}s.", GetEglErrorString());
        return false;
    }
    EGLint count;
    EGLint configAttribs[] = { EGL_SURFACE_TYPE,
                               EGL_WINDOW_BIT,
                               EGL_RED_SIZE,
                               8,
                               EGL_GREEN_SIZE,
                               8,
                               EGL_BLUE_SIZE,
                               8,
                               EGL_ALPHA_SIZE,
                               8,
                               EGL_RENDERABLE_TYPE,
                               EGL_OPENGL_ES3_BIT,
                               EGL_NONE };
    if (eglChooseConfig(eglDisplay_, configAttribs, &config_, 1, &count) == EGL_FALSE) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "EglRenderContext",
                     "EglRenderContext::Init: Failed to bind choose config, error: %{public}s.", GetEglErrorString());
        return false;
    }
    const EGLint contextAttribs[] = { EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE };
    eglContext_ = eglCreateContext(eglDisplay_, config_, EGL_NO_CONTEXT, contextAttribs);
    if (eglContext_ == EGL_NO_CONTEXT) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "EglRenderContext",
                     "EglRenderContext::Init: Failed to bind create context, error: %{public}s.", GetEglErrorString());
        return false;
    }

    OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "EglRenderContext", "EglRenderContext::Init end.");
    return true;
}

  2. Create a NativeImage object, obtain its NativeWindow object by calling OH_NativeImage_AcquireNativeWindow(), and register the callback method for NativeImage.

    bool OpenGLRenderThread::CreateNativeImage()
{
    nativeImage_ = OH_NativeImage_Create(-1, GL_TEXTURE_EXTERNAL_OES);
    if (nativeImage_ == nullptr) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "OpenGLRenderThread", "OH_NativeImage_Create failed.");
        return false;
    }
    int ret = 0;
    {
        std::lock_guard<std::mutex> lock(nativeImageSurfaceIdMutex_);
        nativeImageWindow_ = OH_NativeImage_AcquireNativeWindow(nativeImage_);
        ret = OH_NativeImage_GetSurfaceId(nativeImage_, &nativeImageSurfaceId_);
    }
    if (ret != 0) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "OpenGLRenderThread",
                     "OH_NativeImage_GetSurfaceId failed, ret is %{public}d.", ret);
        return false;
    }

    nativeImageFrameAvailableListener_.context = this;
    nativeImageFrameAvailableListener_.onFrameAvailable = &OpenGLRenderThread::OnNativeImageFrameAvailable;
    ret = OH_NativeImage_SetOnFrameAvailableListener(nativeImage_, nativeImageFrameAvailableListener_);
    if (ret != 0) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "OpenGLRenderThread",
                     "OH_NativeImage_SetOnFrameAvailableListener failed, ret is %{public}d.", ret);
        return false;
    }

    return true;
}

  3. Obtain the NativeWindow object of XComponent from its OnSurfaceCreatedCB callback.

    void OnSurfaceCreatedCB(OH_NativeXComponent *component, void *window)
{
    OH_LOG_Print(LOG_APP, LOG_INFO, LOG_PRINT_DOMAIN, "Callback", "OnSurfaceCreatedCB");
    if ((component == nullptr) || (window == nullptr)) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "Callback",
                     "onSurfaceCreatedCB: component or window is null");
        return;
    }
    char idStr[OH_XCOMPONENT_ID_LEN_MAX + 1] = {'\0'};
    uint64_t idSize = OH_XCOMPONENT_ID_LEN_MAX + 1;
    if (OH_NativeXComponent_GetXComponentId(component, idStr, &idSize) != OH_NATIVEXCOMPONENT_RESULT_SUCCESS) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "Callback",
                     "OnSurfaceCreatedCB:Unable to get XComponent id");
        return;
    }
    std::string id(idStr);
    auto render = PluginRender::GetInstance(id);
    uint64_t width;
    uint64_t height;
    int32_t xSize = OH_NativeXComponent_GetXComponentSize(component, window, &width, &height);
    if ((xSize != OH_NATIVEXCOMPONENT_RESULT_SUCCESS) || (render == nullptr)) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "Callback",
                     "OnSurfaceCreatedCB: Unable to qet XComponent size");
        return;
    }
    render->nativeWindow = reinterpret_cast<OHNativeWindow *>(window);
    (void)OH_NativeWindow_NativeWindowHandleOpt(render->nativeWindow, SET_BUFFER_GEOMETRY, static_cast<int>(width),
                                                    static_cast<int>(height));
    if (id == "OpenGL") {
        render->openGLRenderThread_->UpdateNativeWindow(render->nativeWindow, width, height);
    } else if(id == "Vulkan") {
        render->vulkanRenderThread_->UpdateNativeWindow(render->nativeWindow, width, height);
    }
}

    Call eglCreateWindowSurface() to create an EGLSurface object based on the NativeWindow object of XComponent.

    void OpenGLRenderThread::UpdateNativeWindow(void *window, uint64_t width, uint64_t height)
{
    OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "OpenGLRenderThread", "UpdateNativeWindow.");
    auto nativeWindow = reinterpret_cast<OHNativeWindow *>(window);
    PostTask([this, nativeWindow, width, height](OpenGLRender &renderContext) {
        if (nativeWindow_ != nativeWindow) {
            if (nativeWindow_ != nullptr) {
                (void)OH_NativeWindow_NativeObjectUnreference(nativeWindow_);
            }
            nativeWindow_ = nativeWindow;
            if (nativeWindow_ != nullptr) {
                (void)OH_NativeWindow_NativeObjectReference(nativeWindow_);
            }

            if (eglSurface_ != EGL_NO_SURFACE) {
                renderContext_->DestroyEglSurface(eglSurface_);
                eglSurface_ = EGL_NO_SURFACE;
                CleanGLResources();
            }
        }
        if (nativeWindow_ != nullptr) {
            (void)OH_NativeWindow_NativeWindowHandleOpt(nativeWindow_, SET_BUFFER_GEOMETRY, static_cast<int>(width),
                                                        static_cast<int>(height));
            if (eglSurface_ == EGL_NO_SURFACE) {
                eglSurface_ = renderContext_->CreateEglSurface(static_cast<EGLNativeWindowType>(nativeWindow_));
            }
            if (eglSurface_ == EGL_NO_SURFACE) {
                OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "OpenGLRenderThread", "xfl CreateEglSurface failed.");
                return;
            }
            renderContext_->MakeCurrent(eglSurface_);
            CreateGLResources();
        }
    });
}

  4. Initialize the video decoding environment, including initializing the demuxer and decoder. Note that the decoder is set to the NativeWindow object of NativeImage.

    napi_value PluginRender::StartPlayer(napi_env env, napi_callback_info info)
{
    SampleInfo sampleInfo;
    size_t argc = 4;
    napi_value args[4] = {nullptr};
    napi_get_cb_info(env, info, &argc, args, nullptr, nullptr);
    napi_get_value_int32(env, args[0], &sampleInfo.inputFd);
    napi_get_value_int64(env, args[1], &sampleInfo.inputFileOffset);
    napi_get_value_int64(env, args[2], &sampleInfo.inputFileSize);
    size_t length = 0;
    napi_status status = napi_get_value_string_utf8(env, args[3], nullptr, 0, &length);
    char* buf = new char[length + 1];
    std::memset(buf, 0, length + 1);
    status = napi_get_value_string_utf8(env, args[3], buf, length + 1, &length);
    std::string type = buf;
    PluginRender *render = PluginRender::GetInstance(type);
    if (render != nullptr) {
        if (type == "OpenGL") {
            sampleInfo.window = render->openGLRenderThread_->GetNativeImageWindow();
        } else if (type == "Vulkan") {
            sampleInfo.window = render->vulkanRenderThread_->GetNativeImageWindow();
        } else {
            sampleInfo.window = render->nativeWindow;
        }
    }
    int32_t ret = Player::GetInstance().Init(sampleInfo);
    if (ret != AVCODEC_SAMPLE_ERR_OK) {
        return nullptr;
    }
    Player::GetInstance().Start();
    return nullptr;
}

  5. Start the decoder, decoder input thread, and decoder output thread. (The code snippet is the same as that for XComponent-based rendering.)

  6. Obtain an available AVBuffer through OnNeedInputBuffer, fill the AVBuffer with the video data read by the demuxer in the decoding input thread, and submit it to the decoder for decoding. (The code snippet is the same as that for XComponent-based rendering.)

  7. Obtain the decoded video data through OnNeedOutputBuffer, and submit the data to the output surface in the decoding output thread. (The code snippet is the same as that for XComponent-based rendering.)

  8. Call OH_NativeImage_AttachContext() to bind the context, and then call OH_NativeImage_UpdateSurfaceImage() to update the video image buffer to the OpenGL texture.

    void OpenGLRenderThread::DrawImage()
{
    if (nativeImageTexId_ == 9999) {
        glGenTextures(1, &nativeImageTexId_);
        glBindTexture(GL_TEXTURE_EXTERNAL_OES, nativeImageTexId_);
        // set the texture wrapping parameters
        glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_REPEAT);
        glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_REPEAT);
        // set texture filtering parameters
        glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
        glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    }
    OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "OpenGLRenderThread", "DrawImage.");

    if (eglSurface_ == EGL_NO_SURFACE) {
        OH_LOG_Print(LOG_APP, LOG_WARN, LOG_PRINT_DOMAIN, "OpenGLRenderThread", "eglSurface_ is EGL_NO_SURFACE");
        return;
    }

    OH_NativeImage_AttachContext(nativeImage_, nativeImageTexId_);

    int32_t ret = OH_NativeImage_UpdateSurfaceImage(nativeImage_);
    if (ret != 0) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "OpenGLRenderThread",
                     "OH_NativeImage_UpdateSurfaceImage failed, ret: %{public}d, texId: %{public}d", ret,
                     nativeImageTexId_);
        return;
    }
    OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "OpenGLRenderThread", "OH_NativeImage_UpdateSurfaceImage succeed.");

    ret = OH_NativeImage_GetTransformMatrix(nativeImage_, matrix_);
    if (ret != 0) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "OpenGLRenderThread",
                     "OH_NativeImage_GetTransformMatrix failed, ret: %{public}d", ret);
        return;
    }

    int32_t transformTmp = 2;
    ComputeTransformByMatrix(transformTmp, matrix_);

    DrawVideoImage();
}

  9. Call eglSwapBuffers to submit the rendered buffer to the NativeWindow object of XComponent.

    void OpenGLRender::SwapBuffers(EGLSurface surface) const
{
    EGLBoolean ret = eglSwapBuffers(eglDisplay_, surface);
    if (ret == EGL_FALSE) {
        EGLint surfaceId = -1;
        eglQuerySurface(eglDisplay_, surface, EGL_CONFIG_ID, &surfaceId);
        OH_LOG_Print(
            LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "EglRenderContext",
            "EglRenderContext::SwapBuffers: Failed to SwapBuffers on EGLSurface %{public}d, error is %{public}s.",
            surfaceId, GetEglErrorString());
    }
}

Rendering with Vulkan

[h2]Scenario

Vulkan is a graphics API designed for 2D and 3D rendering, enabling efficient GPU access across platforms. It offers finer-grained hardware control, allowing you to manage resources and states more precisely for improved performance and efficiency. In contrast, OpenGL abstracts certain low-level details from developers, which can sometimes limit performance optimization. However, Vulkan has a significantly steeper learning curve than OpenGL and requires manual management of memory and rendering pipelines. It is recommended for performance-critical scenarios where you already possess Vulkan expertise.

[h2]How to Implement

NativeImage is a module that associates a surface with an OpenGL external texture. It acts as the consumer of the graphics queue and supports two data consumption methods. OpenGL texture binding (introduced in the OpenGL-based rendering section): involves linking data with OpenGL textures and is used within the OpenGL environment. Custom buffer handling: allows you to manually obtain and process buffers. This is the method used for Vulkan-based video rendering.

Once the NativeWindow object of NativeImage is set for the decoder, NativeImage becomes the consumer of the video frame data produced by the decoder. You can call OH_NativeImage_AcquireNativeWindowBuffer() to obtain a NativeWindowBuffer object from NativeImage. This object contains the video frame data. Then, call OH_NativeBuffer_FromNativeWindowBuffer() to convert the OHNativeWindowBuffer object into a NativeBuffer. Finally, submit the NativeBuffer to Vulkan, which creates an ImageView object for rendering. Note that the decoded video frames are usually in YUV format and require conversion for proper rendering.

The specific implementation steps are as follows:

  1. Start the rendering thread, and load the Vulkan dynamic link library.

  2. Create a NativeImage object and obtain its NativeWindow object.

  3. Obtain the NativeWindow object of XComponent, and create a Vulkan surface based on this object.

  4. Initialize the video decoding environment, including initializing the demuxer and decoder.

    Unlike the XComponent-based rendering solution, in the decoder configuration, the surface input parameter is the NativeWindow object of NativeImage, not the NativeWindow object of XComponent.

  5. Start the decoder, decoder input thread, and decoder output thread.

  6. In the decoding input thread, use the demuxer to read video data and submit it to the decoder.

  7. Obtain the decoded video data through OnNeedOutputBuffer, and submit the data to the output surface (the NativeWindow object of NativeImage) in the decoding output thread.

  8. After the NativeImage callback onFrameAvailable() has available data, call OH_NativeImage_AcquireNativeWindowBuffer() to obtain the video data and call OH_NativeBuffer_FromNativeWindowBuffer() to convert the data to the NativeBuffer type.

  9. Create an ImageView in Vulkan from the NativeBuffer for sampling purposes. Create a sampler matching the required format to convert the YUV image into an RGBA image, which is ultimately rendered and displayed through the rendering pipeline.

[h2]How to Develop

  1. Start the rendering thread, and load the Vulkan dynamic link library.

    void VulkanRenderThread::ThreadMainLoop() {
    threadId_ = std::this_thread::get_id();
    if (!InitRenderContext()) {
        return;
    }
    if (!InitNativeVsync()) {
        return;
    }
    if (!CreateNativeImage()) {
        return;
    }
    while (running_) {
        {
            OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread", "Waiting for Vsync.");
            std::unique_lock<std::mutex> Lock(wakeUpMutex_);
            wakeUpCond_.wait(Lock, [this]() { return wakeup_ || vSyncCnt_ > 0 || availableFrameCnt_ > 0; });
            wakeup_ = false;
            vSyncCnt_--;
            (void)OH_NativeVSync_RequestFrame(nativeVsync_, &VulkanRenderThread::OnVsync, this);
        }

        std::vector<VulkanRenderTask> tasks;
        {
            std::lock_guard<std::mutex> lock(taskMutex_);
            tasks.swap(tasks_);
        }
        for (const auto &task : tasks) {
            task(*vulkanRenderContext_);
        }
        if (availableFrameCnt_ <= 0) {
            continue;
        }
        DrawImage();
        availableFrameCnt_--;
    }
}

    Dynamically load libvulkan.so, and load the pointers of the Vulkan base functions.

    bool LoadVulkanLibrary() {
    constexpr char *path = "libvulkan.so";
    dlerror();
    g_libVulkan = dlopen(path, RTLD_NOW | RTLD_LOCAL);
    if (!g_libVulkan) {
        OH_LOG_ERROR(LOG_APP, "Failed to load vulkan Library, %{public}s", dlerror());
        return false;
    }

    vkEnumerateInstanceExtensionProperties = reinterpret_cast<PFN_vkEnumerateInstanceExtensionProperties>(
        dlsym(g_libVulkan, "vkEnumerateInstanceExtensionProperties"));
    vkEnumerateInstanceLayerProperties = reinterpret_cast<PFN_vkEnumerateInstanceLayerProperties>(
        dlsym(g_libVulkan, "vkEnumerateInstanceLayerProperties"));
    vkCreateInstance = reinterpret_cast<PFN_vkCreateInstance>(dlsym(g_libVulkan, "vkCreateInstance"));
    vkGetInstanceProcAddr = reinterpret_cast<PFN_vkGetInstanceProcAddr>(dlsym(g_libVulkan, "vkGetInstanceProcAddr"));
    vkGetDeviceProcAddr = reinterpret_cast<PFN_vkGetDeviceProcAddr>(dlsym(g_libVulkan, "vkGetDeviceProcAddr"));

    return true;
}

  2. Create a NativeImage object and obtain its NativeWindow object.

    bool VulkanRenderThread::CreateNativeImage() {
    nativeImage_ = OH_NativeImage_Create(-1, GL_TEXTURE_EXTERNAL_OES);
    if (nativeImage_ == nullptr) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread", "OH_NativeImage_Create failed.");
        return false;
    }
    int ret = 0;
    {
        std::lock_guard<std::mutex> Lock(nativeImageSurfaceIdMutex_);
        nativeImageWindow_ = OH_NativeImage_AcquireNativeWindow(nativeImage_);
        ret = OH_NativeImage_GetSurfaceId(nativeImage_, &nativeImageSurfaceId_);
    }
    if (ret != 0) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread",
                     "OH_NativeImage_GetSurfaceId failed, ret is %{public}d.", ret);
        return false;
    }

    nativeImageFrameAvailableListener_.context = this;
    nativeImageFrameAvailableListener_.onFrameAvailable = &VulkanRenderThread::OnNativeImageFrameAvailable;
    ret = OH_NativeImage_SetOnFrameAvailableListener(nativeImage_, nativeImageFrameAvailableListener_);
    if (ret != 0) {
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread",
                     "OH_NativeImage_SetonFrameAvailableListener failed, ret is %{public}d.", ret);
        return false;
    }
    return true;
}

  3. Obtain the NativeWindow object of XComponent and create a Vulkan surface based on this object.

    void VulkanRenderThread::UpdateNativeWindow(void *window, uint64_t width, uint64_t height) {
    OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "RenderThread", "UpdateNativeWindow.");
    auto nativeWindow = reinterpret_cast<OHNativeWindow *>(window);
    PostTask([this, nativeWindow, width, height](VulkanRender &renderContext) {
        if (nativeWindow_ != nativeWindow) {
            if (nativeWindow_ != nullptr) {
                (void)OH_NativeWindow_NativeObjectUnreference(nativeWindow_);
            }
            nativeWindow_ = nativeWindow;
            if (nativeWindow_ != nullptr) {
                (void)OH_NativeWindow_NativeObjectReference(nativeWindow_);
            }
        }
        if (nativeWindow_ != nullptr) {
            vulkanRenderContext_->SetupWindow(nativeWindow_);
        }
    });
}

    Set the Vulkan surface and update the Vulkan initialization context, including the Vulkan instance, physical device selection, and Vulkan method loading.

    void VulkanRender::SetupWindow(NativeWindow *nativeWindow) {
    nativeWindow_ = nativeWindow;
    CreateInstance();
    vkExample::utils::LoadVulkanFunctions(instance);
    CreateSurface();
    PickPhysicalDevice();
    CreateLogicalDevice();
    vkExample::utils::LoadVulkanFunctions(device);

    createSwapChain();
    createRenderPass();
    createFrameBuffersAndImages();
    createVertexBuffer();
    createUniformBuffer();
    deviceInfoInitialized = true;
}

    Call vkCreateSurfaceOHOS() to create a surface object.

    bool VulkanRender::CreateSurface() {
    VkSurfaceCreateInfoOHOS surfaceCreateInfo{};
    surfaceCreateInfo.sType = VK_STRUCTURE_TYPE_SURFACE_CREATE_INFO_OHOS;
    if (nativeWindow_ == nullptr) {
        OH_LOG_INFO(LOG_APP, "nativeWindow_ is nulptr.Failed to create surface !");
        return false;
    }
    surfaceCreateInfo.window = nativeWindow_;
    surfaceCreateInfo.flags = 0;
    surfaceCreateInfo.pNext = nullptr;
    bool res = CheckResult(vkCreateSurfaceOHOS(instance, &surfaceCreateInfo, nullptr, &surface));
    return res;
}

  4. Initialize the video decoding environment, including initializing the demuxer and decoder.

    napi_value PluginRender::StartPlayer(napi_env env, napi_callback_info info)
{
    SampleInfo sampleInfo;
    size_t argc = 4;
    napi_value args[4] = {nullptr};
    napi_get_cb_info(env, info, &argc, args, nullptr, nullptr);
    napi_get_value_int32(env, args[0], &sampleInfo.inputFd);
    napi_get_value_int64(env, args[1], &sampleInfo.inputFileOffset);
    napi_get_value_int64(env, args[2], &sampleInfo.inputFileSize);
    size_t length = 0;
    napi_status status = napi_get_value_string_utf8(env, args[3], nullptr, 0, &length);
    char* buf = new char[length + 1];
    std::memset(buf, 0, length + 1);
    status = napi_get_value_string_utf8(env, args[3], buf, length + 1, &length);
    std::string type = buf;
    PluginRender *render = PluginRender::GetInstance(type);
    if (render != nullptr) {
        if (type == "OpenGL") {
            sampleInfo.window = render->openGLRenderThread_->GetNativeImageWindow();
        } else if (type == "Vulkan") {
            sampleInfo.window = render->vulkanRenderThread_->GetNativeImageWindow();
        } else {
            sampleInfo.window = render->nativeWindow;
        }
    }
    int32_t ret = Player::GetInstance().Init(sampleInfo);
    if (ret != AVCODEC_SAMPLE_ERR_OK) {
        return nullptr;
    }
    Player::GetInstance().Start();
    return nullptr;
}

  5. Start the decoder, decoder input thread, and decoder output thread. (The code snippet is the same as that for XComponent-based rendering.)

  6. In the decoding input thread, use the demuxer to read video data and submit it to the decoder. (The code snippet is the same as that for XComponent-based rendering.)

  7. In the decoding output thread, submit the decoded video data to the output surface. (The code snippet is the same as that for XComponent-based rendering.)

  8. After the NativeImage has available data, call OH_NativeImage_AcquireNativeWindowBuffer() to obtain the video data and call OH_NativeBuffer_FromNativeWindowBuffer() to convert the data to the NativeBuffer type.

    void VulkanRenderThread::DrawImage() {
    OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "VulkanRenderThread", "DrawImage.");
    int ret;
    OHNativeWindowBuffer *inBuffer = nullptr;
    OH_NativeBuffer *nativeBuffer = nullptr;
    int32_t fenceFd1 = -1;
    ret = OH_NativeImage_AcquireNativeWindowBuffer(nativeImage_, &inBuffer, &fenceFd1);
    ret = OH_NativeWindow_NativeObjectReference(inBuffer);
    ret = OH_NativeBuffer_FromNativeWindowBuffer(inBuffer, &nativeBuffer);
    if (nativeBuffer == nullptr) {
        OH_NativeWindow_NativeObjectUnreference(inBuffer);
        OH_NativeImage_ReleaseNativeWindowBuffer(nativeImage_, inBuffer, -1);
        OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "VulkanRenderThread", "nativeBuffer is null.");
        return;
    }
    ret = OH_NativeImage_GetTransformMatrix(nativeImage_, matrix_);
    int32_t transformTmp = 0;
    ComputeTransform(transformTmp, matrix_);
    vulkanRenderContext_->hwBufferToTexture(nativeBuffer, matrix_);
    vulkanRenderContext_->render();
    if (lastInBuffer_ != nullptr) {
        OH_NativeWindow_NativeObjectUnreference(lastInBuffer_);
        OH_NativeImage_ReleaseNativeWindowBuffer(nativeImage_, lastInBuffer_, -1);
    }
    lastInBuffer_ = inBuffer;
}

  9. Create a corresponding ImageView in Vulkan for rendering and display based on NativeBuffer, and create a sampler of the corresponding format to convert the YUV format image into an RGBA image for correct rendering and display.

    void VulkanRender::hwBufferToTexture(OH_NativeBuffer *buffer, float transformMatrix[16]) {
    OH_LOG_Print(LOG_APP, LOG_INFO, 0xFF00, "VulkanRenderThread", "hwBufferToTexture.");
    if (!deviceInfoInitialized) {
        return;
    }
    UniformBufferObject ubo{};
    memcpy(ubo.mvp, transformMatrix, sizeof(float) * 16);
    memcpy(buffersInfo.uniformBufferMapped, &ubo, sizeof(ubo));
    VkNativeBufferFormatPropertiesOHOS nbFormatProps = {
        .sType = VK_STRUCTURE_TYPE_NATIVE_BUFFER_FORMAT_PROPERTIES_OHOS,
        .pNext = nullptr
    };
    VkNativeBufferPropertiesOHOS nbProps = {.sType = VK_STRUCTURE_TYPE_NATIVE_BUFFER_PROPERTIES_OHOS,
                                            .pNext = &nbFormatProps};
    CheckResult(vkGetNativeBufferPropertiesOHOS(device, buffer, &nbProps));

    VkImportNativeBufferInfoOHOS importBufferInfo = {
        .sType = VK_STRUCTURE_TYPE_IMPORT_NATIVE_BUFFER_INFO_OHOS,
        .pNext = nullptr,
        .buffer = buffer
    };

    VkMemoryDedicatedAllocateInfo dedicatedAllocateInfo = {
        .sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO,
        .pNext = &importBufferInfo,
        .image = VK_NULL_HANDLE,
        .buffer = VK_NULL_HANDLE
    };

    VkPhysicalDeviceMemoryProperties physicalDeviceMemProps;
    uint32_t foundTypeIndex = 0;
    vkGetPhysicalDeviceMemoryProperties(gpuDevice, &physicalDeviceMemProps);
    uint32_t memTypeCnt = physicalDeviceMemProps.memoryTypeCount;
    for (uint32_t i = 0; i < memTypeCnt; ++i) {
        if (nbProps.memoryTypeBits & (1 << i)) {
            const VkPhysicalDeviceMemoryProperties &pdmp = physicalDeviceMemProps;
            uint32_t supportedFLags = pdmp.memoryTypes[i].propertyFlags & VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
            if (supportedFLags == VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) {
                foundTypeIndex = i;
                break;
            }
        }
    }

    VkMemoryAllocateInfo allocInfo{
        .sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO,
        .pNext = &dedicatedAllocateInfo,
        .allocationSize = nbProps.allocationSize,
        .memoryTypeIndex = 0
    };

    mapMemoryTypeToIndex(nbProps.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT, &allocInfo.memoryTypeIndex);
    VkExternalFormatOHOS externalFormat;
    externalFormat.sType = VK_STRUCTURE_TYPE_EXTERNAL_FORMAT_OHOS;
    externalFormat.pNext = nullptr;
    externalFormat.externalFormat = 0;
    if (nbFormatProps.format == VK_FORMAT_UNDEFINED) {
        externalFormat.externalFormat = nbFormatProps.externalFormat;
    }

    VkExternalMemoryImageCreateInfo externalMemoryImageInfo = {
        .sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO,
        .pNext = &externalFormat,
        .handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_OHOS_NATIVE_BUFFER_BIT_OHOS,
    };

    VkImageUsageFlags usageFlags = VK_IMAGE_USAGE_SAMPLED_BIT;
    if (nbFormatProps.format != VK_FORMAT_UNDEFINED) {
        usageFlags = usageFlags | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT |
                     VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_INPUT_ATTACHMENT_BIT;
    }
    OH_NativeBuffer_Config config;
    OH_NativeBuffer_GetConfig(buffer, &config);
    VkImageCreateInfo image_create_info = {
        .sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO,
        .pNext = &externalMemoryImageInfo,
        .flags = 0,
        .imageType = VK_IMAGE_TYPE_2D,
        .format = nbFormatProps.format,
        .extent = {static_cast<uint32_t>(config.width), static_cast<uint32_t>(config.height), 1},
        .mipLevels = 1,
        .arrayLayers = 1,
        .samples = VK_SAMPLE_COUNT_1_BIT,
        .tiling = VK_IMAGE_TILING_OPTIMAL,
        .usage = usageFlags,
        .sharingMode = VK_SHARING_MODE_EXCLUSIVE,
        .queueFamilyIndexCount = 1,
        .pQueueFamilyIndices = &queueFamilyIndex_,
        .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED
    };

    CheckResult(vkCreateImage(device, &image_create_info, nullptr, &externalTextureImage));
    dedicatedAllocateInfo.image = externalTextureImage;
    CheckResult(vkAllocateMemory(device, &allocInfo, nullptr, &externalTextureMemory));
    CheckResult(vkBindImageMemory(device, externalTextureImage, externalTextureMemory, 0));

    VkSamplerYcbcrConversionCreateInfo ycbcrCreateInfo = {
        .sType = VK_STRUCTURE_TYPE_SAMPLER_YCBCR_CONVERSION_CREATE_INFO,
        .ycbcrRange = nbFormatProps.suggestedYcbcrRange,
        .components = nbFormatProps.samplerYcbcrConversionComponents,
        .xChromaOffset = nbFormatProps.suggestedXChromaOffset,
        .yChromaOffset = nbFormatProps.suggestedYChromaOffset,
        .chromaFilter = VK_FILTER_LINEAR,
        .forceExplicitReconstruction = false
    };

    if (nbFormatProps.format == VK_FORMAT_UNDEFINED) {
        ycbcrCreateInfo.pNext = &externalFormat;
        ycbcrCreateInfo.format = VK_FORMAT_UNDEFINED;
        ycbcrCreateInfo.ycbcrModel = nbFormatProps.suggestedYcbcrModel;
    }

    CheckResult(
        vkCreateSamplerYcbcrConversion(device, &ycbcrCreateInfo, nullptr, &externalTextureInfo.ycbcrConversion));

    VkSamplerYcbcrConversionInfo convInfoWrap = {
        .sType = VK_STRUCTURE_TYPE_SAMPLER_YCBCR_CONVERSION_INFO,
        .conversion = externalTextureInfo.ycbcrConversion,
        .pNext = nullptr,
    };

    VkImageViewCreateInfo view = {
        .sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO,
        .pNext = &convInfoWrap,
        .flags = 0,
        .image = externalTextureImage,
        .viewType = VK_IMAGE_VIEW_TYPE_2D,
        .format = nbFormatProps.format,
        .components =
            {
                VK_COMPONENT_SWIZZLE_IDENTITY,
                VK_COMPONENT_SWIZZLE_IDENTITY,
                VK_COMPONENT_SWIZZLE_IDENTITY,
                VK_COMPONENT_SWIZZLE_IDENTITY
            },
        .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1},
    };
    CheckResult(vkCreateImageView(device, &view, nullptr, &externalTextureInfo.view));

    const VkSamplerCreateInfo sampler = {
        .sType = VK_STRUCTURE_TYPE_SAMPLER_CREATE_INFO,
        .pNext = &convInfoWrap,
        .magFilter = VK_FILTER_NEAREST,
        .minFilter = VK_FILTER_NEAREST,
        .mipmapMode = VK_SAMPLER_MIPMAP_MODE_NEAREST,
        .addressModeU = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE,
        .addressModeV = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE,
        .addressModeW = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE,
        .mipLodBias = 0.0f,
        .compareEnable = VK_FALSE,
        .anisotropyEnable = VK_FALSE,
        .maxAnisotropy = 1,
        .compareOp = VK_COMPARE_OP_NEVER,
        .minLod = 0.0f,
        .maxLod = 0.0f,
        .borderColor = VK_BORDER_COLOR_FLOAT_OPAQUE_WHITE,
        .unnormalizedCoordinates = VK_FALSE
    };
    CheckResult(vkCreateSampler(device, &sampler, nullptr, &externalTextureInfo.sampler));

    createDescriptorSetLayout();
    createDescriptorSet();

    VkDescriptorImageInfo imageInfo = {
        .sampler = externalTextureInfo.sampler,
        .imageView = externalTextureInfo.view,
        .imageLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL
    };

    VkDescriptorBufferInfo bufferInfo = {
        .buffer = buffersInfo.uniformBuf, .offset = 0, .range = sizeof(UniformBufferObject)};
    VkWriteDescriptorSet bufferWrite = {.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET,
                                        .dstSet = gfxPipelineInfo.descSet,
                                        .dstBinding = 0,
                                        .dstArrayElement = 0,
                                        .descriptorCount = 1,
                                        .descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER,
                                        .pImageInfo = nullptr,
                                        .pBufferInfo = &bufferInfo,
                                        .pTexelBufferView = nullptr};
    VkWriteDescriptorSet imageWrite = {.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET,
                                       .dstSet = gfxPipelineInfo.descSet,
                                       .dstBinding = 1,
                                       .dstArrayElement = 0,
                                       .descriptorCount = 1,
                                       .descriptorType = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER,
                                       .pImageInfo = &imageInfo,
                                       .pBufferInfo = nullptr,
                                       .pTexelBufferView = nullptr};
    gfxPipelineInfo.descWrites[0] = bufferWrite;
    gfxPipelineInfo.descWrites[1] = imageWrite;
    vkUpdateDescriptorSets(device, 2, gfxPipelineInfo.descWrites, 0, nullptr);

    createGraphicsPipeline();
    createOtherStaff();

    recordCommandBuffer();
    OH_LOG_INFO(LOG_APP, "hwBufferToTexture end");
}

Sample Code