目录

Image Decoding Memory Optimization (ArkTS)

When an app performs image decoding, it needs to allocate the corresponding memory. The memory usage depends on the memory allocation type and pixel format. This guide describes different memory types, pixel formats, and how to combine them for optimal decoding performance.

The app side obtains the PixelMap through the decoding API and passes it to the Image component for display.

When the PixelMap is large and uses shared memory, the RS main thread will experience a longer texture upload time, leading to lag. The graphics side provides a DMA memory zero-copy feature, which avoids texture upload time consumption when drawing images. In addition, setting an appropriate pixel format (such as YUV) can further reduce memory usage.

Memory Types

The memory types for the PixelMap are as follows:

The system provides the createPixelMapUsingAllocator API, allowing you to customize the memory allocation type for decoding. For details about the API definition and usage examples, see the image decoding API description Interface (ImageSource).

[h2]Differences Between SHARE_MEMORY and DMA_ALLOC

Item SHARE_MEMORY DMA_ALLOC
Definition Shared memory (such as ashmem/anonymous sharing) provided by the operating system, allowing multiple processes to read/write the same physical pages. Buffers allocated for direct DMA access by peripherals, GPU, or display pipelines. Common implementations include dmabuf and SurfaceBuffer, designed for zero-copy operations.
Working principle Processes access a shared memory region via the CPU. Transferring this data to GPU or display pipelines typically requires a copy. Data is written directly to the buffer (for example, by a decoder via DMA), which the GPU or display pipelines can then access without any copying.
Use case Inter-process or inter-thread data sharing (for example, exchanging intermediate results in algorithms or post-processing). High-bandwidth data transfer scenarios such as hardware decoding of images/videos, camera preview, and display rendering.
CPU usage The CPU is involved in managing and synchronizing shared memory (for example, locking and unlocking), which incurs additional overhead. Extremely low CPU usage: The CPU is only involved in the configuration of the DMA controller, with no intervention in actual data transfer.
Hardware dependency Dependent on the shared memory mechanism of the operating system. Strongly dependent on the hardware DMA controller.
Memory allocation and access permissions The system allocates physical or virtual memory areas for shared memory, with access requiring user or kernel mapping operations. The DMA controller directly operates the physical memory, requiring pre-allocated DMA buffers (usually contiguous memory).
Advantages High flexibility. Supports simultaneous data sharing by multiple threads or processes, facilitating image post-processing and collaboration. High efficiency and low latency, suitable for transfer of large data volumes and continuous data blocks.
Disadvantages Shared memory operations require additional synchronization mechanisms, increasing programming complexity and CPU load. Hardware support is required, and the data transfer range is limited by DMA address space (usually requiring contiguous physical memory).

[h2]Advantages of Using DMA_ALLOC Memory

When using DMA_ALLOC, you must pay attention to the difference between the stride and the image width, and perform alignment before data reading, parsing, and display.

[h2]Restrictions

The current image decoding feature has the following restrictions on memory allocation modes:

When createPixelMapUsingAllocator is used for decoding, if the specified memory allocation mode does not match the image format or decoding method, an exception indicating a memory allocation failure is thrown.

If the allocation type is set to AUTO, the system determines whether to use DMA_ALLOC or SHARE_MEMORY based on the decoding and rendering time.

Different memory allocation strategies can result in differences in the stride of the image. For memory allocated by using DMA_ALLOC, the stride must be used for operations such as editing on the PixelMap. The following describes how to obtain the stride.

[h2]Obtaining the Stride

The stride describes the storage width of each row of pixel data of an image in memory. It is an important parameter in the image drawing process and is used to correctly locate the layout of the image data in the memory.

When memory is allocated using DMA_ALLOC, the stride must meet the hardware alignment requirements.

The stride value can be obtained by calling getImageInfo().

  1. Call getImageInfo() to obtain an ImageInfo object.

  2. Access the stride value (info.stride) from the ImageInfo object.

    // Import the related modules.
import { image } from '@kit.ImageKit';
import { common } from '@kit.AbilityKit';
    async CreatePixelMapUsingAllocator(context: Context, type: image.AllocatorType): Promise<image.PixelMap | undefined> {
  const resourceMgr = context.resourceManager;
  try {
    const rawFile = await resourceMgr.getRawFileContent('99_132.jpg'); // The test image is a 99*132 JPG image.
    let imageSource: image.ImageSource = image.createImageSource(rawFile.buffer as ArrayBuffer);
    let options: image.DecodingOptions = {};
    let pixelmap = await imageSource.createPixelMapUsingAllocator(options, type);
    if (pixelmap != undefined) {
      let info = await pixelmap.getImageInfo();
      // The stride of the pixelmap allocated with DMA_ALLOC memory differs from that of the pixelmap allocated with SHARE_MEMORY memory.
      console.info('stride = ' + info.stride);
    }
    return pixelmap;
  } catch (err) {
    console.error(`Create PixelMap by setting allocator type failed: ${err}.`);
    return undefined;
  }
    
}

Pixel Format

The pixel format after image decoding directly affects memory usage. The main supported pixel formats are as follows.

[h2]Differences Between RGBA_8888 and YUV Formats

Name RGBA_8888 NV21/NV12 (YUV 4:2:0)
Definition Color information consists of four components: R (Red), G (Green), B (Blue), and Alpha (transparency). Each component occupies 8 bits, totaling 32 bits. Color information consists of a luma component Y and interleaved chroma components UV. The Y component occupies 8 bits, and the UV components average 4 bits due to 4:2:0 sampling, totaling an average of 12 bits.
Bytes per Pixel 4 bytes Approximately 1.5 bytes
Memory Usage Calculation Width × height × 4 Width × height × 1.5
Applicable Scenarios Scenarios requiring Alpha channel processing, such as transparency compositing and shadow effects. Image preview, display, and similar scenarios. Low memory usage, suitable for large-size image decoding.
Advantages Supports full Alpha channel operations with good compatibility. Low memory usage. JPEG hardware decoding can directly output this format, avoiding format conversion overhead.

[h2]Advantages of Using YUV Format

  • SVG and TIFF format images do not support decoding to YUV pixel format.
  • The YUV format does not include an Alpha channel. For images requiring transparency, use the RGBA_8888 format.

[h2]Setting the YUV Pixel Format

When decoding in YUV format, set the desiredPixelFormat parameter in DecodingOptions. It is recommended to use DMA memory allocation together with this setting.

async CreatePixelMapWithYUV(context: Context): Promise<image.PixelMap | undefined> {
  const resourceMgr = context.resourceManager;
  try {
    const rawFileDescriptor = await resourceMgr.getRawFd('test.jpeg');
    let imageSource: image.ImageSource = image.createImageSource(rawFileDescriptor);

    // Set the YUV pixel format and DMA memory allocation for optimal decoding performance.
    let options: image.DecodingOptions = {
      desiredPixelFormat: image.PixelMapFormat.NV21  // Set to YUV format. NV12 is also available.
    };
    let pixelmap = await imageSource.createPixelMapUsingAllocator(options, image.AllocatorType.DMA);

    if (pixelmap != undefined) {
      let info = await pixelmap.getImageInfo();
      console.info('YUV PixelMap created, stride = ' + info.stride + ', pixelFormat = ' + info.pixelFormat);
    }
    return pixelmap;
  } catch (err) {
    console.error(`Create PixelMap with YUV format failed: ${err}.`);
    return undefined;
  }
}

System Default Memory Allocation

When decoding with createPixelMap or createPixelMapSync, the system automatically selects shared memory or DMA memory.

To specify the memory type, call createPixelMapUsingAllocator or createPixelMapUsingAllocatorSync, and set allocatorType to image.AllocatorType.DMA or image.AllocatorType.SHARE_MEMORY.

Memory Restrictions for Decoding a Single Image

To prevent system crashes from memory overflow, the system enforces memory restrictions on processes. For details, see Application-Killed Issues Detection.

The image framework imposes a 2 GB memory limit for decoding a single image. Processes should actively manage their memory usage. You are advised to release PixelMap objects promptly when they are no longer needed, to avoid process termination by the system.

Applications can use onMemoryLevel to listen for system memory changes.

The calculation rule for PixelMap memory allocation is as follows:

pixels_size (pixel memory size) = stride (image pixel storage width) * height (image pixel height)

For images with original pixel memory exceeding 2 GB that support downsampling, you are advised to use the createPixelMap or createPixelMapUsingAllocator API, and set desiredSize (expected output size) in DecodingOptions for downsampling decoding.

Starting from API version 21, for images that support downsampling decoding, when desiredSize (expected output size) is set, the decoder calculates PixelMap pixel memory at the optimal downsampling rate with a base gradient of 1/8. This means that it selects the highest clarity sampling rate among 7/8, 6/8, ..., 1/8.

The table below lists the downsampling decoding support for different image formats in the image framework.

Support for Downsampling Image Format
Supported .jpg, .png, .heic12+ (Refer to the device specification document for specific support.)
Not supported .gif .bmp .webp .dng .svg10+ .ico11+