目录

Single-Operator Application

Overview

CANN Kit provides independent operator creation and computation pathways. Third-party frameworks can migrate operators like convolution and depthwise convolution to the NPU through a single-operator interfacing method during model loading and inference. Thanks to accelerated computing by the hardware platform, this approach offers greater flexibility compared to network-wide mode and delivers better performance than network-wide CPU computation.

The following describes the APIs for creating a single-operator tensor, and creating, loading, and executing a single-operator executor. For details about how to use these APIs, please refer to Development Procedure. For more setting and query APIs, see API Reference.

Table 1 Single-operator APIs and functions

API Description
HiAI_SingleOpTensorDesc * HMS_HiAISingleOpTensorDesc_Create (const int64_t *dims, size_t dimNum, HiAI_SingleOpDataType dataType, HiAI_SingleOpFormat format, bool isVirtual); Creates a HiAI_SingleOpTensorDesc object.
void HMS_HiAISingleOpTensorDesc_Destroy (HiAI_SingleOpTensorDesc **tensorDesc); Releases a HiAI_SingleOpTensorDesc object.
HiAI_SingleOpBuffer * HMS_HiAISingleOpBuffer_Create (size_t dataSize); Creates a HiAI_SingleOpBuffer object based on the specified memory size.
size_t HMS_HiAISingleOpBuffer_GetSize (const HiAI_SingleOpBuffer *buffer); Queries the byte size of HiAI_SingleOpBuffer.
void * HMS_HiAISingleOpBuffer_GetData (const HiAI_SingleOpBuffer *buffer); Queries the memory address of HiAI_SingleOpBuffer.
OH_NN_ReturnCode HMS_HiAISingleOpBuffer_Destroy (HiAI_SingleOpBuffer **buffer); Releases a HiAI_SingleOpBuffer object.
HiAI_SingleOpTensor * HMS_HiAISingleOpTensor_CreateFromTensorDesc (const HiAI_SingleOpTensorDesc *desc); Creates a HiAI_SingleOpTensor object based on HiAI_SingleOpTensorDesc.
HiAI_SingleOpTensor * HMS_HiAISingleOpTensor_CreateFromConst (const HiAI_SingleOpTensorDesc *desc, void *data, size_t dataSize); Creates a HiAI_SingleOpTensor object based on HiAI_SingleOpTensorDesc and the memory address and data size of constant data (such as convolution weight and bias).
HiAI_SingleOpTensorDesc * HMS_HiAISingleOpTensor_GetTensorDesc (const HiAI_SingleOpTensor *tensor); Obtains the tensor description of HiAI_SingleOpTensor.
HiAI_SingleOpBuffer * HMS_HiAISingleOpTensor_GetBuffer (const HiAI_SingleOpTensor *tensor); Obtains the buffer of HiAI_SingleOpTensor.
OH_NN_ReturnCode HMS_HiAISingleOpTensor_Destroy (HiAI_SingleOpTensor **tensor); Releases a HiAI_SingleOpTensor object.
HiAI_SingleOpOptions * HMS_HiAISingleOpOptions_Create (void); Creates a HiAI_SingleOpOptions object.
void HMS_HiAISingleOpOptions_Destroy (HiAI_SingleOpOptions **options); Releases a HiAI_SingleOpOptions object.
HiAI_SingleOpDescriptor* HMS_HiAISingleOpDescriptor_CreateConvolution(HiAISingleOpDescriptor_ConvolutionParam param); Creates a descriptor object for a convolution class (common convolution, transposed convolution, or depthwise convolution).
void HMS_HiAISingleOpDescriptor_Destroy (HiAI_SingleOpDescriptor **opDesc); Releases a HiAI_SingleOpDescriptor object.
HiAI_SingleOpExecutor* HMS_HiAISingleOpExecutor_CreateConvolution(HiAI_SingleOpExecutorConvolutionParam param); Creates a HiAI_SingleOpExecutor object for a convolution operator.
size_t HMS_HiAISingleOpExecutor_GetWorkspaceSize (const HiAI_SingleOpExecutor *executor); Obtains the size (in bytes) of the ION memory workspace required by HiAI_SingleOpExecutor.
OH_NN_ReturnCode HMS_HiAISingleOpExecutor_Init (HiAI_SingleOpExecutor *executor, void *workspace, size_t workspaceSize); Loads HiAI_SingleOpExecutor.
OH_NN_ReturnCode HMS_HiAISingleOpExecutor_Execute (HiAI_SingleOpExecutor *executor, HiAI_SingleOpTensor *input[], int32_t inputNum, HiAI_SingleOpTensor *output[], int32_t outputNum); Executes synchronous computational inference.
OH_NN_ReturnCode HMS_HiAISingleOpExecutor_Destroy (HiAI_SingleOpExecutor **executor); Destroys HiAI_SingleOpExecutor object to release the memory occupied by the executor.

Development Procedure

The following uses a single convolution operator as an example.

  1. Create a single-operator executor.

    1. Call HMS_HiAISingleOpOptions_Create to create a single-operator configuration object.

    2. Call HMS_HiAISingleOpDescriptor_CreateConvolution to create a convolution operator descriptor object.

    3. Call HMS_HiAISingleOpTensor_CreateFromConst to create the weight and bias tensors of the convolution operator.

    4. Call HMS_HiAISingleOpTensorDesc_Create to create the description objects of the single-operator input and output tensors.

    5. Call HMS_HiAISingleOpExecutor_CreateConvolution to create a single-operator executor, by passing the convolution operator's descriptor object, weight tensor, bias tensor, as well as the input tensor description and output tensor description.

      To create a fusion operator executor for the convolution operator and activation operator, call HMS_HiAISingleOpDescriptor_CreateActivation to create a descriptor object for the activation operator, then call HMS_HiAISingleOpExecutor_CreateFusedConvolutionActivation to create a fusion operator executor.

    6. After the creation is successful, call HMS_HiAISingleOpDescriptor_Destroy to destroy the operator descriptor object and call HMS_HiAISingleOpOptions_Destroy to destroy the configuration object for creating a single operator.

  2. Create an input/output tensor.

    1. Call HMS_HiAISingleOpTensor_CreateFromTensorDesc to create the single-operator input and output tensors.
    2. After the creation is successful, call HMS_HiAISingleOpTensorDesc_Destroy to destroy the tensor descriptor object.
    3. Call HMS_HiAISingleOpTensor_GetBuffer to obtain the internal buffer object of the input/output tensor.
    4. Call HMS_HiAISingleOpBuffer_GetData to obtain the allocated input/output ION memory address. The address can be used to write the input and read the output of a single operator during network-wide model inference.

  3. Load the single-operator executor.

    1. Call HMS_HiAISingleOpExecutor_GetWorkspaceSize to obtain the size of the ION memory workspace required by the created single-operator executor for inference computing.
    2. Call HMS_HiAISingleOpBuffer_Create to create sufficient workspace based on the size of the ION memory workspace required by the single-operator executor.
    3. Call HMS_HiAISingleOpBuffer_GetData to obtain the address of the allocated ION memory workspace.
    4. Call HMS_HiAISingleOpExecutor_Init to load the created single-operator executor using the workspace memory address and workspace size.

  4. Perform the inference.

    Call HMS_HiAISingleOpExecutor_Execute to perform synchronous inference.

  5. Uninstall the single-operator executor to release resources.

Example

Assuming there is a deep convolution operator with input dimensions of 1x8x224x224 and float32 input data arranged in NCHW format. After preparing the weight and bias data in NCHW format, you can refer to the following sample code for using the single-operator API to perform inference and obtain the float32 output in NCHW format:

// Example operator parameters
// Single-operator convolution mode.
HiAI_SingleOpConvMode convMode = HIAI_SINGLEOP_CONV_MODE_DEPTHWISE;
int64_t strides[2] = {1, 1};
int64_t dilations[2] = {1, 1};
int64_t pads[4] = {0, 0, 0, 0};
int64_t groups = 1;
// Single-operator padding mode.
HiAI_SingleOpPadMode padMode = HIAI_SINGLEOP_PAD_MODE_SAME;
int64_t filterDims[4] = {8, 1, 3, 3};
size_t filterDataSize = 8 * 1 * 3 * 3 * sizeof(float);
void *filterData = malloc(filterDataSize);
// ...
int64_t biasDims[1] = {8};
size_t biasDataSize = 8 * sizeof(float);
void *biasData = malloc(biasDataSize);
// ...
int64_t inputDims[4] = {1, 8, 224, 224};
HiAI_SingleOpDataType inputDataType = HIAI_SINGLEOP_DT_FLOAT;
// Single operator tensor layout format.
HiAI_SingleOpFormat inputFormat = HIAI_SINGLEOP_FORMAT_NCHW;
bool inputIsVirtual = false;
// If the operator's output data type and layout format are not specified, set the data type to HIAI_SINGLEOP_DT_UNDEFINED and the layout format to HIAI_SINGLEOP_FORMAT_RESERVED.
// After creating a single operator, call HMS_HiAISingleOpExecutor_UpdateOutputTensorDesc to update the output tensor description to the optimal data type and layout format suited for the hardware.
int64_t outputDims[4] = {1, 8, 224, 224};
HiAI_SingleOpDataType outputDataType = HIAI_SINGLEOP_DT_FLOAT;
HiAI_SingleOpFormat outputFormat = HIAI_SINGLEOP_FORMAT_NCHW;
bool outputIsVirtual = false;

// Create a single-operator executor.
options_ = HMS_HiAISingleOpOptions_Create();
HiAISingleOpDescriptor_ConvolutionParam convOpDescCreateParam = {convMode, {0}, {0}, {0}, groups, padMode};
memcpy(convOpDescCreateParam.strides, strides, opDescSize * sizeof(int64_t));
memcpy(convOpDescCreateParam.dilations, dilations, opDescSize * sizeof(int64_t));
memcpy(convOpDescCreateParam.pads, pads, tensorSize * sizeof(int64_t));
// Create a descriptor object for the convolution class.
convOpDesc_ = HMS_HiAISingleOpDescriptor_CreateConvolution(convOpDescCreateParam);
// Create a single-operator tensor descriptor object based on the dimensions, data type, and format.
filterDesc_ = HMS_HiAISingleOpTensorDesc_Create(filterDims, tensorSize, inputDataType, inputFormat, false);
// Create a single operator tensor object.
filter_ = HMS_HiAISingleOpTensor_CreateFromConst(filterDesc_, filterData, filterDataSize);
biasDesc_ = HMS_HiAISingleOpTensorDesc_Create(biasDims, 1, outputDataType, outputFormat, false);
bias_ = HMS_HiAISingleOpTensor_CreateFromConst(biasDesc_, biasData, biasDataSize);
inputDesc_ = HMS_HiAISingleOpTensorDesc_Create(inputDims, tensorSize, inputDataType, inputFormat, inputIsVirtual);
outputDesc_ = HMS_HiAISingleOpTensorDesc_Create(outputDims, tensorSize, outputDataType, outputFormat,
                                                outputIsVirtual);
// Construct the parameters for the single operator convolution executor.
executorCreateParam_ = {options_, convOpDesc_, inputDesc_, outputDesc_, filter_, bias_};
// ...
// Create the single-operator convolution executor.
executor_ = HMS_HiAISingleOpExecutor_CreateConvolution(executorCreateParam_);
if (executor_ == nullptr) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp executor create failed");
    // ...
}
// It is recommended that unnecessary resources be destroyed in a timely manner.
HMS_HiAISingleOpTensorDesc_Destroy(&filterDesc_);
HMS_HiAISingleOpTensorDesc_Destroy(&biasDesc_);
HMS_HiAISingleOpOptions_Destroy(&options_);
HMS_HiAISingleOpDescriptor_Destroy(&convOpDesc_);
ret = HMS_HiAISingleOpTensor_Destroy(&filter_);
if (ret != OH_NN_SUCCESS) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp filter destroy failed");
    // ...
}
ret = HMS_HiAISingleOpTensor_Destroy(&bias_);
if (ret != OH_NN_SUCCESS) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp bias destroy failed");
    // ...
}
// ...
// Calculate the time consumed for operator graph construction.
std::chrono::system_clock::time_point createTimeBegin = std::chrono::system_clock::now();
// Create an input/output tensor.
input_ = HMS_HiAISingleOpTensor_CreateFromTensorDesc(inputDesc_);
output_ = HMS_HiAISingleOpTensor_CreateFromTensorDesc(outputDesc_);
// The input and output tensors of the single operator must use ION memory to reduce copy overhead.
// After the input tensor is successfully created, you can use the following method to obtain the ION memory address in the input tensor for input data filling.
// This method also applies for obtaining the ION memory address in the output tensor, which is used to read the output data after the inference and computation are successful.
HiAI_SingleOpBuffer *inputBuffer = HMS_HiAISingleOpTensor_GetBuffer(input_);
void *inputData = HMS_HiAISingleOpBuffer_GetData(inputBuffer);
size_t inputDataSize = HMS_HiAISingleOpBuffer_GetSize(inputBuffer);
memset(inputData, 0, inputDataSize);
std::chrono::system_clock::time_point createTimeEnd = std::chrono::system_clock::now();
createTensorTime = GetRunTime(createTimeBegin, createTimeEnd);

HMS_HiAISingleOpTensorDesc_Destroy(&inputDesc_);
HMS_HiAISingleOpTensorDesc_Destroy(&outputDesc_);
// ...
// Obtain the size (in bytes) of the ION memory workspace required for the single-operator executor.
size_t workspaceSize = HMS_HiAISingleOpExecutor_GetWorkspaceSize(executor_);
// If there are multiple single-operator executors, the workspace memory of each executor can be reused. You only need to apply for the maximum workspace required.
workspaceBuffer_ = HMS_HiAISingleOpBuffer_Create(workspaceSize);
void *workspace = HMS_HiAISingleOpBuffer_GetData(workspaceBuffer_);
// Before calling this API, allocate the workspace memory required by the executor.
OH_NN_ReturnCode ret = HMS_HiAISingleOpExecutor_Init(executor_, workspace, workspaceSize);
if (ret != OH_NN_SUCCESS) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp executor init failed");
    // ...
}
// ...
// Perform the inference operation for multiple rounds and collect statistics on the inference duration of the operator.
for (size_t i = 0; i < times; i++) {
    for (size_t j = 0; j < opNum; j++) {
        // Perform the inference.
        HiAI_SingleOpTensor *inputs[] = {input_};
        HiAI_SingleOpTensor *outputs[] = {output_};
        std::chrono::system_clock::time_point executeTimeBegin = std::chrono::system_clock::now();
        OH_NN_ReturnCode ret = HMS_HiAISingleOpExecutor_Execute(executor_, inputs, 1, outputs, 1);
        if (ret != OH_NN_SUCCESS) {
            OH_LOG_ERROR(LOG_APP, "HMS_HiAISingleOp executor execute failed");
            // ...
        }
        std::chrono::system_clock::time_point executeTimeEnd = std::chrono::system_clock::now();
        auto executeElapsedTime = GetRunTime(executeTimeBegin, executeTimeEnd);
        OH_LOG_INFO(LOG_APP, "idx-%zu execute succ: %llu us", j, executeElapsedTime);
        aveTime[j] += executeElapsedTime;
    }
    OH_LOG_INFO(LOG_APP, "------ Round %zu ------ ", i);
}
// ...
// Calculate the time consumed for releasing operator resources.
std::chrono::system_clock::time_point destroyTimeBegin = std::chrono::system_clock::now();
// Destroy the input tensor and free resources.
OH_NN_ReturnCode ret = HMS_HiAISingleOpTensor_Destroy(&input_);
if (ret != OH_NN_SUCCESS) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp input_ destroy failed");
    // ...
}
// Destroy the output tensor and free resources.
ret = HMS_HiAISingleOpTensor_Destroy(&output_);
if (ret != OH_NN_SUCCESS) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp output_ destroy failed");
    // ...
}
// Destroy the single-operator buffer object.
ret = HMS_HiAISingleOpBuffer_Destroy(&workspaceBuffer_);
if (ret != OH_NN_SUCCESS) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp workspaceBuffer_ destroy failed");
    // ...
}
// Destroy the single-operator executor and release the memory occupied by the executor.
ret = HMS_HiAISingleOpExecutor_Destroy(&executor_);
if (ret != OH_NN_SUCCESS) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp executor destroy failed");
    // ...
}
std::chrono::system_clock::time_point destroyTimeEnd = std::chrono::system_clock::now();
destroyTime = GetRunTime(destroyTimeBegin, destroyTimeEnd);
// ...
// Summarize the time consumption of each phase.
std::vector<float> outputs(returnArraySize, 0);
if (!times_) {
    OH_LOG_ERROR(LOG_APP, "iteration times_ is not initialized or is zero");
    return outputs;
}
if (aveTime.empty()) {
    OH_LOG_INFO(LOG_APP, "HMS_HiAISingleOp_GetResult failed");
    return outputs;
}
// Obtain the graph construction time and convert microseconds to seconds.
createTensorTime /= 1000000.0f;
outputs[0] = createTensorTime;
// Obtain the inference time.
executeTime = 0;
for (size_t i = 0; i < opNum; i++) {
    aveTime[i] /= times_;
    OH_LOG_INFO(LOG_APP, "idx-%zu average time: %.2f us", i, aveTime[i]);
    executeTime += aveTime[i];
}
OH_LOG_INFO(LOG_APP, "op average time sum: %.2f us", executeTime);
executeTime /= 1000000.0f;
outputs[1] = executeTime;
// Obtain the resource release time.
destroyTime /= 1000000.0f;
outputs[2] = destroyTime;
OH_LOG_INFO(LOG_APP, "GetResult success");
return outputs;