Instruction queues executed on the AI Core include the vector instruction queue (V), matrix instruction queue (M), and storage and movement instruction queues (MTE2 and MTE3). Different instruction queues can be executed independently and in parallel. This is the cornerstone of double buffering optimization.
The CopyIn and CopyOut processes use the storage moving instruction queue (MTE2 and MTE3), and the Compute process uses the Vector instruction queue (V), which means that the CopyIn and CopyOut processes and the Compute process can be performed concurrently.
Figure 1 shows a complete CopyIn, Compute, and CopyOut process. During the CopyIn phase, data is moved from the Global Memory to the Local Memory. After the Vector Unit completes the computation, the result is moved back to the Global Memory through the CopyOut process.
Figure 1 Data movement and vector compute
In this process, data movement and vector compute are performed in serial mode. Therefore, the Vector Unit is idle during data movement. For example, if each of the three phases (CopyIn, Compute, and CopyOut) takes the time t, respectively. In this case, the time utilization ratio of the Vector Unit is only 1/3, resulting in long waiting time and low resource utilization.
To reduce the waiting time of the Vector Unit, the double buffering mechanism divides the data to be processed into two parts, for example, Tensor1 and Tensor2. As shown in Figure 2, when Vector computes the data in Tensor1, Tensor2 can perform the CopyIn process. When Vector computes the data in Tensor2, Tensor1 can perform the CopyOut process. In this way, data movement and vector compute are performed in parallel, the utilization ratio of the Vector Unit is greatly improved.
To sum up, double buffering hides the data movement time and reduces the wait time between MTE/Vector instructions and Vector compute process, thereby improving the Vector utilization. You can set the number of memory blocks when allocating memory for a queue to implement data parallelism. The following is a simple code example:
pipe.InitBuffer(inQueueX, 2, 256);
Figure 2 Double buffer mechanism
In most cases, the double buffering mechanism can effectively improve the utilization ratio of vector compute time and reduce the operator execution time. However, the double buffering mechanism does not always result in higher overall performance. Example:
If the data movement time accounts only a small part of the total time, the double buffering mechanism brings relatively smaller performance gain.
If the volume of data to be processed is small enough that the Vector Unit can compute all data at once, double buffering will bring no gain. Instead, it will reduce the utilization ratio of the Vector Unit.
Therefore, before determining whether to use double buffering, consider the Vector Unit compute capability, data volume, and proportion of the movement time and compute process time.