目录

Model Inference

Basic Concepts

Model inference includes model compilation and inference, which is the basics for other extended and enhanced functions.

Service Process

The following figure shows the model inference process:

API Description

The following APIs are the main process APIs. For more compilation, loading, and execution configurations, see API Reference.

Table 1 APIs in CANN Kit for model inference

API Description
OH_NNCompilation* OH_NNCompilation_ConstructWithOfflineModelBuffer(const void *modelBuffer, size_t modelSize); Creates a model compilation instance based on the model buffer.
OH_NN_ReturnCode OH_NNCompilation_SetDevice(OH_NNCompilation *compilation, size_t deviceID); Sets the target device for model compilation and execution.
OH_NN_ReturnCode OH_NNCompilation_Build(OH_NNCompilation *compilation); Compiles the model. The generated model is saved in compilation.
OH_NNExecutor* OH_NNExecutor_Construct(OH_NNCompilation *compilation); Creates an executor for model inference based on the compiled model.
NN_Tensor* OH_NNTensor_Create(size_t deviceID, NN_TensorDesc* tensorDesc); Construct the input and output tensors.
OH_NN_ReturnCode OH_NNExecutor_RunSync(OH_NNExecutor *executor, NN_Tensor *inputTensor[], size_t inputCount, NN_Tensor *outputTensor[], size_t outputCount); Performs synchronous model inference.
void OH_NNCompilation_Destroy(OH_NNCompilation **compilation); Destroys the model compilation instance.
OH_NN_ReturnCode OH_NNTensor_Destroy(NN_Tensor** tensor); Destroys the input and output tensors.
void OH_NNExecutor_Destroy(OH_NNExecutor **executor); Destroys the executor of model inference.

Development Procedure

The following describes the main development procedure for model inference. For details, see SampleCode.

  1. Prepare the model and development environment.

    • Prepare the offline model (OM model), which can be generated using the tools_omg tool or obtained from Model Zoo.
    • Download and configure the DevEco Studio environment for developing and debugging a HarmonyOS app.

  2. Create a DevEco Studio project.

  3. Create a model compilation instance.

    Call OH_NNCompilation_ConstructWithOfflineModelBuffer to read the model buffer and create a model compilation instance. Alternatively, call OH_NNCompilation_ConstructWithOfflineModelFile to directly read the model file and create a model compilation instance.

  4. Select the target device.

    Call OH_NNDevice_GetAllDevicesID to obtain all device IDs, search for the device ID whose name is HIAI_F, record the ID, and set it to the compilation instance created in step 3 using OH_NNCompilation_SetDevice.

  5. Compile the model.

    Call OH_NNCompilation_Build and pass the model compilation instance created in step 3 to compile the model. The compiled model data is still stored in the model compilation instance.

  6. Create a model executor.

    Call OH_NNExecutor_Construct to create an executor instance for the compiled model. After the executor is created, call OH_NNCompilation_Destroy to destroy the model compilation instance.

  7. Construct the input and output tensors.

    Call OH_NNExecutor_GetInputCount to query the number of inputs, obtain the TensorDesc of the corresponding index by using OH_NNExecutor_CreateInputTensorDesc, and create a tensor by using OH_NNTensor_Create based on the TensorDesc. Then, you can write actual data to the tensor. Repeat these steps to construct output tensors.

  8. Perform model inference.

    Call OH_NNExecutor_RunSync to perform synchronous inference of the model. The output data of the model is stored in outputTensors. Process the output data as required to obtain the expected result.

  9. Destroy the instance.