A user enters a game scene suitable for frame generation.
The game app calls the HMS_FG_CreateContext_VK API to create a frame generation context instance. If the frame generation context instance fails to be created, the operation of providing current frame information in step 6 does not need to be performed. Instead, just render and display real frames one by one.
The game app calls APIs to set properties of the frame generation context instance. For example, the game app calls the HMS_FG_SetAlgorithmMode_VK (mandatory) API to set the frame generation algorithm mode (selecting the interpolation mode) and calls other related APIs as required.
The game sets the integration mode on the system side.

By default, the game display mode is selected for the integration mode on the system side. If the system display mode is selected, you can call HMS_FG_SetIntegrationMode_VK to set the integration information for frame generation FG_IntegrationInfo. For details about the display mode, see FG_PresentMode.

In system display mode, you can call HMS_FG_SetUiPredictionEnabled_VK to enable the UI prediction function. If this function is not enabled, the predicted frame will reuse the UI of the previous frame for display.

In system display mode, developers can call HMS_FG_SetTargetFps_VK to set the target frame rate after frame generation. If this API is not called, the target frame rate is set to 60 fps by default.
The game app calls the HMS_FG_Activate_VK API to activate the frame generation context instance.
The game app renders a real frame and then calls the HMS_FG_Dispatch_VK API, passing the color, depth, and camera matrix information about the real frame, to generate a predicted frame.
The game app completes UI drawing and sends the real frame for display.
The user exits the game scene suitable for frame generation.
The game app calls the HMS_FG_DestroyContext_VK API to destroy the frame generation context instance and release memory resources.

Development Procedure

The following describes an example of using the Vulkan graphics API for frame generation, where the system sends predicted frames for display. For details about the code, see Graphics Development Sample (Frame Generation with Vulkan).

Configure the metadata. Declare metadata in the module.json5 file of the app to support the system display mode.

{
  "module": {
    // ...
    "metadata": [
      {
        "name": "GraphicsAccelerateKit_FusionAware",
        "value": "Vulkan"
      },
      // ...
    ],
    // ...
  }
}

Include the Graphics Accelerate Kit frame generation header file frame_generation_vk.h.

// Include the frame generation header file frame_generation_vk.h.
#include <graphics_game_sdk/frame_generation_vk.h>

Call the HMS_FG_CreateContext_VK API to create a frame generation context instance. If nullptr is returned, it indicates that the frame generation context instance fails to be created or the current device does not support the frame generation function.

 // Variable declaration.
 VkInstance vkInstance = VK_NULL_HANDLE; // vkInstance is created by calling vkCreateInstance.
 VkPhysicalDevice vkPhysicalDevice = VK_NULL_HANDLE; // vkPhysicalDevice is enumerated by calling vkEnumeratePhysicalDevices.
 VkDevice vkDevice = VK_NULL_HANDLE; // vkDevice is created by calling vkCreateDevice.

 // Create a frame generation context instance.
FG_ContextDescription_VK contextDescription{};
contextDescription.vkInstance = vkInstance;
contextDescription.vkPhysicalDevice = vkPhysicalDevice;
contextDescription.vkDevice = vkDevice;
contextDescription.framesInFlight = 1;
contextDescription.fnVulkanLoaderFunction = vkGetInstanceProcAddr;
FG_Context_VK* m_context = HMS_FG_CreateContext_VK(&contextDescription);
if (m_context == nullptr) {
    GOLOGE("HMS_FG_CreateContext_VK execution failed.");
    return false;
}

Call APIs to set properties of the context instance. Set the frame generation algorithm mode to interpolation and select the system display mode.

// Initialize the error code of frame generation API calls.
FG_ErrorCode errorCode = FG_SUCCESS;

// Frame generation algorithm mode.
FG_AlgorithmModeInfo aInfo{};
aInfo.predictionMode = FG_PREDICTION_MODE_INTERPOLATION; // Interpolation mode
aInfo.meMode = FG_ME_MODE_BASIC; // Basic motion estimation mode.

errorCode = HMS_FG_SetAlgorithmMode_VK(m_context, &aInfo); // Set the frame generation algorithm mode.
if (errorCode != FG_SUCCESS) {
    GOLOGE("HMS_FG_SetAlgorithmMode_VK execution failed, error code: %d.", errorCode);
    return false;
}

// Call other related APIs for configuration.
// ...

// Integration information for predicted frame generation.
FG_IntegrationInfo integrationInfo {};
integrationInfo.presentMode = FG_PRESENT_BY_SYSTEM; // Mode for sending predicted frames for display.
integrationInfo.textureCachedByGame = false; // Indicates whether the game buffers the input color texture and depth texture. If yes, the system will not create additional copies for prediction. The default value is false.
integrationInfo.needFlipInputColor = false; // Indicates whether the color texture needs to be flipped. The default value is false.
integrationInfo.needFlipOutputColor = false; // Indicates whether the predicted frame needs to be flipped. The default value is false.
// Set the integration information for predicted frame generation.
errorCode = HMS_FG_SetIntegrationMode_VK(m_context, &integrationInfo);
if (errorCode != FG_SUCCESS) {
    GOLOGE("HMS_FG_SetIntegrationMode_VK execution failed, error code: %d.", errorCode);
    return false;
}

// If the color buffer is flipped 180 degrees around the Y axis relative to the depth stencil buffer, set the second parameter to true. If this API is not called, the default value false is used.
errorCode = HMS_FG_SetDepthStencilYDirectionInverted_VK(m_context, false);
if (errorCode != FG_SUCCESS) {
    GOLOGE("HMS_FG_SetDepthStencilYDirectionInverted_VK execution failed, error code: %d.", errorCode);
    return false;
}

// Set the target frame rate for frame generation. The setting takes effect only for the system display mode and after the game is released. It is invalid for the game display mode. If this API is not called, the frame rate is not limited by default but depends on the game rendering frame rate.
errorCode = HMS_FG_SetTargetFps_VK(m_context, 60);
if (errorCode != FG_SUCCESS) {
    GOLOGE("HMS_FG_SetTargetFps_VK execution failed, error code: %d.", errorCode);
    return false;
}

Call the HMS_FG_Activate_VK API to activate the frame generation context instance.

// Activate the frame generation context instance.
FG_ErrorCode errorCode = HMS_FG_Activate_VK(m_context);
if (errorCode != FG_SUCCESS) {
    GOLOGE("HMS_FG_Activate_VK execution failed, error code: %d.", errorCode);
    // ...
    return false;
}

Call the HMS_FG_CreateImage_VK API to create an image instance for the real frame color buffer and an image instance for the depth stencil buffer.

// Variable declaration.
FG_Image_VK *m_ffSceneColor = nullptr;
VulkanFG::Image m_sceneColor{};
FG_Image_VK *m_ffDepthStencil = nullptr;
VulkanFG::Image m_sceneDepthStencil{};

// Create an image instance for the real frame color buffer.
m_ffSceneColor = HMS_FG_CreateImage_VK(m_context, m_sceneColor.GetNativeImage(), m_sceneColor.GetNativeImageView());
if (!m_ffSceneColor) {
    GOLOGE("HMS_FG_CreateImage_VK m_ffSceneColor execution failed.");
    return false;
}
// Create an image instance for the real frame depth stencil buffer.
m_ffDepthStencil = HMS_FG_CreateImage_VK(m_context, m_sceneDepthStencil.GetNativeImage(),
                                         m_sceneDepthStencil.GetNativeImageView());
if (!m_ffDepthStencil) {
    GOLOGE("HMS_FG_CreateImage_VK m_ffDepthStencil execution failed.");
    return false;
}

During game running, buffer the color, depth, and camera matrix information for real frame rendering. During predicted frame rendering, call the HMS_FG_Dispatch_VK API, passing the properties of the real frame and specifying the predicted frame buffer index, to generate a predicted frame. The game sends the current real frame for display, while the system sends the predicted frame for display between the current real frame and the previous real frame.

// Variable declaration.
FG_Mat4x4 m_viewProj{};
FG_Mat4x4 m_invViewProj{};
FG_DispatchDescription_VK dispatch{};

// Real frame rendering phase.
// Render the game scene to obtain the current real frame. Buffer its color, depth, and camera matrix information for generating the next predicted frame and drawing the real frame.
// ...

// Draw the UI.
// ...

bool const runPrediction = m_predictionEnabled & !m_predictionPaused;
if (runPrediction) { // Predicted frame rendering phase.
    dispatch = {
        // Pass the property information about the real frame color buffer.
        .inputColorInfo = {
            .image = m_ffSceneColor,
            // Set the synchronization status of the real frame color buffer before the predicted frame is generated.
            .initialSync {
                .accessMask = VK_ACCESS_SHADER_READ_BIT,
                .layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,
                .stages = VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT
            },
            // Set the synchronization status of the real frame color buffer after the predicted frame is generated.
            .finalSync {
                .accessMask = VK_ACCESS_SHADER_READ_BIT,
                .layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,
                .stages = VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT
            }
        },
        // Pass the property information about the real frame depth stencil buffer.
        .inputDepthStencilInfo = {
            .image = m_ffDepthStencil,
            // Set the synchronization status of the depth stencil buffer before the predicted frame is generated.
            .initialSync {
                .accessMask = VK_ACCESS_SHADER_READ_BIT,
                .layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,
                .stages = VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT
            },
            // Set the synchronization status of the depth stencil buffer after the predicted frame is generated.
            .finalSync {
                .accessMask = VK_ACCESS_SHADER_READ_BIT,
                .layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,
                .stages = VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT
            }
        },
        // Pass the property information about the predicted frame buffer.
        .outputColorInfo = {
            .image = m_ffPredictedColor,
            // Set the synchronization status of the predicted frame buffer before the predicted frame is generated.
            .initialSync {
                .accessMask = VK_ACCESS_SHADER_READ_BIT,
                .layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,
                .stages = VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT
            },
            // Set the synchronization status of the predicted frame buffer after the predicted frame is generated.
            .finalSync {
                .accessMask = VK_ACCESS_SHADER_READ_BIT,
                .layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,
                .stages = VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT
            }
        },
        // Pass the view projection matrix of the previous real frame.
        .viewProj = m_viewProj,
        // Pass the inverse view projection matrix of the previous real frame.
        .invViewProj = m_invViewProj,
        // Pass the handle of a command buffer used to record frame drawing commands.
        .vkCommandBuffer = fif->commandBuffer,
        // Pass the sequence number of the current frame.
        .frameIdx = fifIndex
    };
    // ...
    // Generate a predicted frame and update the memory of the predicted frame buffer.
    FG_ErrorCode const errorCode = HMS_FG_Dispatch_VK(m_context, &dispatch);
    if (errorCode != FG_SUCCESS) {
        GOLOGE("HMS_FG_Dispatch_VK execution failed, error code: %d", errorCode);
    }
}
    
// ...
// Send the real frame for display.
// ...

Call the HMS_FG_DestroyContext_VK API to destroy the frame generation context instance and release memory resources.

// Destroy the frame generation context instance and release memory resources.
errorCode = HMS_FG_DestroyContext_VK(&m_context);
if (errorCode != FG_SUCCESS) {
    GOLOGE("HMS_FG_DestroyContext_VK execution failed, error code: %d", errorCode);
    return false;
}

OpenGL ES

AI-based Frame Generation Mode

Send feedback

Was this page helpful?

Send feedback

Reach out to us through the community for any other questions.

Community Q&A Intelligent Assistant