目录

llm_engine.h

Overview

This file defines APIs for LLM model inference.

File to include: <CANNKit/llm_engine.h>

Library: libcann_llm_engine.so

System capability: SystemCapability.AI.CANN.LLMEngine

Since: 6.1.1(24)

Related module: CANN

Summary

[h2]Types

Name Description
typedef struct HMS_LLMEngine_Context HMS_LLMEngine_Context Defines the alias of the LLM engine context.
typedef struct HMS_LLMEngine_Executor HMS_LLMEngine_Executor Defines the LLM engine executor.
typedef struct HMS_LLMEngine_Prompt HMS_LLMEngine_Prompt Defines the text input for the LLM engine.
typedef void(* callbackFunctionType) (const HMS_LLMEngine_Context *) Defines the generation callback function.

[h2]Enums

Name Description

HMS_LLMEngine_InferPerfMode {

HMS_LLMENGINE_INFERPERF_UNSET = 0, HMS_LLMENGINE_INFERPERF_LOW, HMS_LLMENGINE_INFERPERF_MIDDLE, HMS_LLMENGINE_INFERPERF_HIGH,

HMS_LLMENGINE_INFERPERF_EXTREME_HIGH

}

 Enumerates inference performance modes.

[h2]Functions

Name Description
HMS_LLMEngine_Executor * HMS_LLMEngineExecutor_CreateFromExecutorJson (const char *jsonFile) Creates an LLM engine executor handle using a JSON configuration file.
void HMS_LLMEngine_Context_Destroy (HMS_LLMEngine_Context **ctx) Destroys the LLM engine context.
HMS_LLMEngine_Context * HMS_LLMEngineContext_CreateFromContextJson (const char *jsonFile) Creates an LLM engine context handle using a JSON configuration file.
void HMS_LLMEngineExecutor_Destroy (HMS_LLMEngine_Executor **executor) Destroys an LLM engine executor, deallocating its memory.
HMS_LLMEngine_Prompt * HMS_LLMEnginePrompt_Create (void) Creates an LLM engine prompt handle.
OH_NN_ReturnCode HMS_LLMEnginePrompt_SetText (HMS_LLMEngine_Prompt *prompt, const char *text) Sets the text input.
OH_NN_ReturnCode HMS_LLMEnginePrompt_SetTokenId (HMS_LLMEngine_Prompt *prompt, int32_t *tokenIds, uint32_t tokenNum) Sets the input token IDs.
void HMS_LLMEnginePrompt_Destroy (HMS_LLMEngine_Prompt **prompt) Destroys the LLM engine prompt handle.
OH_NN_ReturnCode HMS_LLMEngineContext_SetOnOneTokenGenerateDoneFunc (HMS_LLMEngine_Context *ctx, callbackFunctionType func) Sets the callback function triggered during token generation.
OH_NN_ReturnCode HMS_LLMEngineContext_SetOnAllTokensGenerateDoneFunc (HMS_LLMEngine_Context *ctx, callbackFunctionType func) Sets the callback function triggered when all tokens are fully generated.
OH_NN_ReturnCode HMS_LLMEngineContext_SetOnGenerateAsyncFailed (HMS_LLMEngine_Context *ctx, callbackFunctionType func) Sets the callback function triggered upon generation failure.
OH_NN_ReturnCode HMS_LLMEngineContext_GetOneGenerationLen (const HMS_LLMEngine_Context *ctx, uint32_t *len) Obtains the length of the generated text chunk.
OH_NN_ReturnCode HMS_LLMEngineContext_GetOneGeneration (const HMS_LLMEngine_Context *ctx, char *generation, uint32_t len) Obtains the generated text chunk.
OH_NN_ReturnCode HMS_LLMEngineContext_GetAllGenerationLen (const HMS_LLMEngine_Context *ctx, uint32_t *len) Obtains the total length of all generated text.
OH_NN_ReturnCode HMS_LLMEngineContext_GetAllGeneration (const HMS_LLMEngine_Context *ctx, char *generation, uint32_t len) Obtains all generated text.
OH_NN_ReturnCode HMS_LLMEngineContext_GetOneTokenGeneration (const HMS_LLMEngine_Context *ctx, int32_t *genToken) Obtains the generated token ID.
OH_NN_ReturnCode HMS_LLMEngineContext_GetAllTokenGenerationLen (const HMS_LLMEngine_Context *ctx, uint32_t *len) Obtains the length of all generated token IDs.
OH_NN_ReturnCode HMS_LLMEngineContext_GetAllTokenGeneration (const HMS_LLMEngine_Context *ctx, int32_t *genToken, uint32_t len) Obtains all generated token IDs.
OH_NN_ReturnCode HMS_LLMEngineExecutor_Generate (HMS_LLMEngine_Executor *executor, HMS_LLMEngine_Context *ctx, const HMS_LLMEngine_Prompt *prompt) Executes synchronous LLM inference.
OH_NN_ReturnCode HMS_LLMEngineExecutor_GenerateAsync (HMS_LLMEngine_Executor *executor, HMS_LLMEngine_Context *ctx, const HMS_LLMEngine_Prompt *prompt) Executes asynchronous LLM inference.
OH_NN_ReturnCode HMS_LLMEngineExecutor_SetInferencePerfMode (HMS_LLMEngine_Executor *executor, HMS_LLMEngine_InferPerfMode inferPerfMode) Sets the inference performance mode.
OH_NN_ReturnCode HMS_LLMEngineContext_GetTotalTimeMs (const HMS_LLMEngine_Context *ctx, double *ms) Obtains the total consumed time, in ms.
OH_NN_ReturnCode HMS_LLMEngineContext_GetPrefillTimeMs (const HMS_LLMEngine_Context *ctx, double *ms) Obtains the prefill time, in ms.
OH_NN_ReturnCode HMS_LLMEngineContext_GetDecodeTimeMs (const HMS_LLMEngine_Context *ctx, double *ms) Obtains the decoding time, in ms.
OH_NN_ReturnCode HMS_LLMEngineContext_GetInputTokenCount (const HMS_LLMEngine_Context *ctx, uint64_t *count) Obtains the number of input tokens.
OH_NN_ReturnCode HMS_LLMEngineContext_GetOutputTokenCount (const HMS_LLMEngine_Context *ctx, uint64_t *count) Obtains the number of output tokens.