目录

Matrix Programming Operator Implementation

Implementation Process

The previous section describes the data tiling solution and data flows for Matmul matrix multiplication. Ascend C provides a group of Matmul high-level APIs that encapsulate the common algorithm logic for tiling, data movement, and computation, helping you quickly implement Matmul matrix multiplication operations. You can call APIs on the host to automatically obtain tiling parameters. After the parameter is passed to the kernel upon initialization, the matrix multiplication operations can be completed through several simple APIs. The following code contains only the key steps of Matmul and cannot be run directly.

To enable a host to automatically obtain tiling parameters, perform the following steps:

  1. Create a tiling object.

    auto ascendcPlatform = platform_ascendc::PlatformAscendC(context->GetPlatformInfo());
matmul_tiling::MatmulApiTiling cubeTiling(ascendcPlatform);

    When creating an object, you need to pass the hardware platform information, which can be obtained by calling GetPlatformInfo.

  2. Set the data types and formats of A, B, and bias.

    cubeTiling.SetAType(AscendC::TPosition::GM, CubeFormat::ND, matmul_tiling::DataType::DT_FLOAT16);
cubeTiling.SetBType(AscendC::TPosition::GM, CubeFormat::ND, matmul_tiling::DataType::DT_FLOAT16);
cubeTiling.SetCType(AscendC::TPosition::GM, CubeFormat::ND, matmul_tiling::DataType::DT_FLOAT);
cubeTiling.SetBiasType(AscendC::TPosition::GM, CubeFormat::ND, matmul_tiling::DataType::DT_FLOAT);

  3. Set the matrix shape information.

    cubeTiling.SetShape(M, N, K);
cubeTiling.SetOrgShape(M, N, K);

  4. Set the available space size.

    cubeTiling.SetBufferSpace(-1, -1, -1);

  5. Set other parameters as required, for example, bias that will participate in the compute.

    cubeTiling.SetBias(true);

  6. Obtains tiling parameters.

    MatmulCustomTilingData tiling;
if (cubeTiling.GetTiling(tiling.cubeTilingData) == -1){
    return ge::GRAPH_FAILED;
}

  7. Perform other operations such as serialization and saving of tiling parameters.

To use Matmul APIs to implement matrix multiplication, perform the following steps:

  1. Create a Matmul object. An example is as follows:

    #include "lib/matmul_intf.h"
typedef matmul::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> aType;
typedef matmul::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> bType;
typedef matmul::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> cType;
typedef matmul::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> biasType;
matmul::Matmul<aType, bType, cType, biasType> mm;

    During object creation, input the type information of parameters A, B, C, and Bias. The type information is defined by MatmulType, including the logical location of memory, data format, and data type.

  2. Performs initialization.

    mm.Init(&tiling.cubeTilingData, &pipe); // initialization

  3. Set the left matrix A, right matrix B, and bias.

    mm.SetTensorA(gm_a); // Set the left cube A.
mm.SetTensorB(gm_b); // Set the right cube B.
mm.SetBias(gm_bias); // Set the bias.

  4. Execute the matrix multiplication.

    • Call Iterate to complete a single iterative computation, and use a while loop to compute the full data on a single core. The Iterate method allows for flexible control over the number of iterations required to compute the desired amount of data.
      while (mm.Iterate()) {
     mm.GetTensorC(gm_c);
 }
    • Call IterateAll to compute all data on a single core. The IterateAll method does not require cyclic iterations and is relatively simple to use.
      mm.IterateAll(gm_c);

  5. End the matrix multiplication.

    mm.End();

Setting the Format

During Matmul object creation, input the type information of parameters A, B, C, and Bias. The type information is defined by MatmulType, including the logical memory location, data format, and data type. An example is as follows:

typedef matmul::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> aType;
typedef matmul::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, half> bType;
typedef matmul::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> cType;
typedef matmul::MatmulType<AscendC::TPosition::GM, CubeFormat::ND, float> biasType;
matmul::Matmul<aType, bType, cType, biasType> mm;

Data formats include CubeFormat::ND, CubeFormat::NZ, and CubeFormat::ND_ALIGN. For details about the ND and NZ formats, see Data Formats.

ND_ALIGN is used to configure the output matrix based on certain padding rules. ND to ND_ALIGN conversion is shown in the following figure. The matrix data type is uint32_t. Assume that the output matrix is output to the UB and the N direction of the original matrix is not 32-byte aligned. Pad 0s to align ND_ALIGN to 32 bytes.

Setting Shape Information

Shape information can be set during host tiling for tiling compute. Some shape information can also be modified when the kernel is running for scenarios such as tail block setting and Matmul reuse (multiple Matmul computations reuse one Matmul object). This section describes the shape concepts involved and provides guidance on how to set the tiling information on the host and kernel.

Based on the introduction to Data Tiling, we have learned the concepts of orgShape (M, N, K), singleCoreShape (singleCoreM, singleCoreN, singleCoreK), and baseShape (baseM, baseN, baseK), as shown in the following figure.

In addition, during single-core Matmul tiling, the shape that actually participates in Matmul computation can be a part of the original shape. singleM, singleN, and singleK express the shape that actually participates in Matmul computation, as shown in the following figure. In single-core scenarios, singleM, singleN, and singleK are passed through singleCoreM, singleCoreN, and singleCoreK.