Multi-channel recall uses diverse strategies or algorithms to quickly filter the candidate result set from massive data. These strategies or algorithms can be used for different features, models, or data sources, aiming to cover various potential scenarios. ArkData provides the inverted index and vector recall capabilities and supports flexible conditional filtering.
Re-ranking performs secondary filtering on records in the candidate result set obtained through multi-channel recall. It calculates the relevance score of each result through simple rules or using more complex models (such as machine learning or deep learning models) and re-ranks the results.

Inverted Index Recall

Inverted index recall includes two steps: selection of core terms and definition of inverted index recall sub-strategies.

[h2]Selection of Core Terms

Upon a user query, the set of core terms must be identified first by filtering based on the term weight, part-of-speech, and relative position, prior to constructing the SQL query for the inverted index. During re-ranking, whether the core terms are matched is a key indicator for assessing the relevance of the candidate results. The current logic for selecting core terms is as follows:

Traverse the tokenized results from beginning to end and accumulate the weights of the tokens until the cumulative weight exceeds (the total weight of all tokens * core term coefficient). The tokens accumulated up to this point are considered the core terms.
Any token whose part-of-speech is eng (English word or letter) or m (number) is automatically treated as a core term.

The token weights and part-of-speech information in the query are derived from the tokenization results of the GaussDB tokenizer.

[h2]Definition of Inverted Index Recall Sub-strategies

To ensure the accuracy of the recall results, the inverted index recall uses the SQLite FTS5 inverted index engine and provides three sub-strategies, namely, three inverted index operators for matching and scoring related results. The inverted index operators are as follows:

BM25 operator: calculates the final score based on the weighted sum of BM25 scores for query term matches across various fields.
Exact match operator: assigns a score if the document field contains an exact match of the query phrase.
Unordered match operator: assigns a score if all user query terms appear within a specified window size in the document field.

Additionally, each inverted index operator provides matching features. The final inverted index recall score is the weighted result of the three strategies.

Vector Recall

Vector recall works by first converting a user query into a vector (using an embedding model for vectorization) and then retrieving similar vectors, thereby recalling semantically relevant content. The similarity threshold for vector approximation is configured in the recall settings.

Reverse Lookup

After multi-channel recall is completed, some results are retrieved only by vector recall (not by inverted index recall) or only by inverted index recall (not by vector recall). During multi-channel recall result aggregation, these differential results lack recall information from the other channel. Since the re-ranking module requires each recall result to contain information from both channels, a reverse lookup operation must be performed on these differential results.

Ranking Module

The ranking module involves tiering the results and performing in-tier ranking, utilizing algorithms such as Reciprocal Rank Fusion (RRF) and score fusion ranking.

[h2]Result Tiering

Multi-channel recall results are tiered by relevance based on the feature values or recall scores for each channel. There are three tiers, which are high, medium, and low, for the service side to determine the relevance of the final retrieval results.

For inverted index recall, tiering is determined by how well the matched fields in the query results hit the user's query terms.

High: The user's query terms are exactly matched in one or more fields.
Medium: Most core terms are matched.
Low: Only some query terms are matched.

"Most core terms are matched" is defined as follows: If the number of core terms is less than or equal to 3, all core terms must be matched. If the number of core terms is greater than 3, N–1 core terms must be matched, where N indicates the total number of core terms identified in the query.

For vector recall, tiering is performed based on one or more configured vector score thresholds. A document is assigned to a tier if its vector score meets or exceeds the threshold for that tier. A vector score threshold consists of one or two numbers in the range [0, 1]. If there are two vector score thresholds, the two values indicate the high-tier threshold and medium-tier threshold, respectively, and a vector score below the medium-tier threshold falls into the low tier. If there is only one vector score threshold, the value indicates the high-tier threshold, and a vector score below this threshold falls into the low tier (no medium-tier threshold in this case).

If the tiering results of inverted index recall and vector recall are different for the same retrieval result, the higher tier takes precedence.

[h2]In-Tier Ranking Algorithm: RRF

The multi-channel recall results are re-ranked based on the RRF algorithm, and weight configuration is supported for each recall channel. The RRF score is calculated based on an item's rank across different recall strategies. For example, if an item has ranks r1, r2, ..., and r**n in different recall strategies, its RRF score can be calculated using the following formula:

In the formula, k is a constant used to adjust the influence of the ranking position. By calculating the RRF score for each item, the algorithm ranks the items based on their scores to generate the final result list.

[h2]In-Tier Ranking Algorithm: Score Fusion Ranking

Multi-channel recall results are re-ranked by calculating the ranking scores based on the recall scores from each channel. Configurable weights are supported for each recall channel. A final score is calculated for each item by calculating a weighted average of its scores across all channels according to the configured weights. Then the items are ranked based on their final scores to generate the result list.

Constraints

Currently, only recall based on vector databases and inverted index databases is supported.
The query length cannot exceed 512 characters.

API Description

The following table describes the APIs related to multi-channel recall and re-ranking.

API	Description
retrieval.getRetriever(config: RetrievalConfig): Promise<Retriever>	Creates and obtains a retriever.
retrieveRdb(query: string, condition: RetrievalCondition): Promise<RdbRecords>	Retrieves data that meets the retrieval conditions (including query tokens, recall conditions, and re-ranking strategies).

Development Procedure

Import the required modules.

import { retrieval } from '@kit.DataAugmentationKit';
import { relationalStore } from '@kit.ArkData';
import { BusinessError } from '@kit.BasicServicesKit';

Initialize the inverted index data and vector data.

let rdbStoreInvIdx: relationalStore.RdbStore;

let rdbStoreVector: relationalStore.RdbStore;

let storeConfigVector: relationalStore.StoreConfig = {
  name: 'vector_test.db',
  securityLevel: relationalStore.SecurityLevel.S3,
  vector: true,
};

let storeConfigInvIdx: relationalStore.StoreConfig = {
  name: 'files_test.db',
  securityLevel: relationalStore.SecurityLevel.S3,
  tokenizer: relationalStore.Tokenizer.CUSTOM_TOKENIZER
};

async initDBData() {
  let context: Context | undefined = this.getUIContext().getHostContext();
  if (context == undefined) {
    console.error('getHostContext failed.');
    return;
  }
  rdbStoreVector = await relationalStore.getRdbStore(context, storeConfigVector);
  if (rdbStoreVector != undefined) {
    let createSql = 'CREATE TABLE IF NOT EXISTS vector (fileid TEXT PRIMARY KEY, filename_text TEXT, filename FLOATVECTOR(128), keywords_text TEXT, keywords FLOATVECTOR(128), chapter FLOATVECTOR(128), abstract FLOATVECTOR(128), int64_value BIGINT, double_value DOUBLE, bool_value BOOLEAN, blob_value BLOB)';
    await rdbStoreVector.execute(createSql);
    this.insertVectorDB();

    let createSql2 = 'CREATE TABLE IF NOT EXISTS keyword_vector (KEYWORD TEXT PRIMARY KEY, EMBEDDING FLOATVECTOR(128))';
    await rdbStoreVector.execute(createSql2);
    this.insertVectorDB2();
  }
  rdbStoreInvIdx = await relationalStore.getRdbStore(context, storeConfigInvIdx);
  if (rdbStoreInvIdx != undefined) {
    let createSql = 'CREATE VIRTUAL TABLE IF NOT EXISTS files USING fts5(fileid, filename, keywords, chapter, abstract, content, tokenize = "customtokenizer")';
    await rdbStoreInvIdx.execute(createSql);
    this.insertInvIdxDB();
  }
  console.info('InitDBData success.');
}

async insertVectorDB2() {
  let sqlInsertKeywordVector1:string =
    'INSERT INTO keyword_vector (keyword, embedding) VALUES ("Sports live streaming scenario", "[0.006954,-0.079041,0.046173,0.157959,-0.017212,0.037018,-0.072083,-0.028488,-0.099854,0.044037,-0.008911,-0.063049,0.035950,-0.105835,0.057739,0.060364,-0.062042,0.044159,0.143188,0.123901,-0.069641,-0.061920,-0.086731,-0.092468,0.092957,-0.027649,-0.005497,-0.039276,0.017502,-0.046570,-0.115906,0.081177,-0.153931,-0.040588,0.123474,-0.099060,0.062042,0.026352,-0.041382,-0.099548,0.071167,-0.120850,0.082642,0.026398,-0.035614,-0.008545,-0.076660,-0.031067,0.192017,-0.052582,0.005310,0.052734,0.199463,0.075195,-0.070740,-0.035950,0.073120,0.089172,0.075989,0.003582,0.050201,-0.012787,0.016647,-0.053619,0.001906,-0.060181,-0.068359,-0.114502,-0.045013,0.004547,-0.004673,-0.148071,0.126343,0.019394,-0.063110,-0.055908,0.071228,0.002369,0.041412,0.126709,-0.053467,0.127808,0.055420,0.206177,0.002169,-0.001452,0.095520,-0.042511,0.099243,-0.164185,0.093384,-0.014618,-0.129150,-0.238770,-0.085327,0.051300,-0.020004,0.010063,-0.084351,-0.003567,0.064941,-0.205322,-0.158936,-0.074768,0.104370,0.197021,-0.080688,-0.066772,-0.036346,0.034912,-0.019760,0.110474,0.128662,0.094727,0.024948,-0.033356,-0.081848,0.054474,-0.065857,-0.156494,0.002527,0.097595,-0.027420,0.039185,0.063965,0.220093,0.029556,-0.115417]");';
  await rdbStoreVector.execute(sqlInsertKeywordVector1);
  // ...
}

async insertVectorDB() {
  let sqlInsertVector1:string =
    'INSERT INTO vector (fileid, filename_text, filename, keywords_text, keywords, chapter, abstract, int64_value, double_value, bool_value, blob_value) VALUES ("0", "Live sports scenario", "[0.006954,-0.079041,0.046173,0.157959,-0.017212,0.037018,-0.072083,-0.028488,-0.099854,0.044037,-0.008911,-0.063049,0.035950,-0.105835,0.057739,0.060364,-0.062042,0.044159,0.143188,0.123901,-0.069641,-0.061920,-0.086731,-0.092468,0.092957,-0.027649,-0.005497,-0.039276,0.017502,-0.046570,-0.115906,0.081177,-0.153931,-0.040588,0.123474,-0.099060,0.062042,0.026352,-0.041382,-0.099548,0.071167,-0.120850,0.082642,0.026398,-0.035614,-0.008545,-0.076660,-0.031067,0.192017,-0.052582,0.005310,0.052734,0.199463,0.075195,-0.070740,-0.035950,0.073120,0.089172,0.075989,0.003582,0.050201,-0.012787,0.016647,-0.053619,0.001906,-0.060181,-0.068359,-0.114502,-0.045013,0.004547,-0.004673,-0.148071,0.126343,0.019394,-0.063110,-0.055908,0.071228,0.002369,0.041412,0.126709,-0.053467,0.127808,0.055420,0.206177,0.002169,-0.001452,0.095520,-0.042511,0.099243,-0.164185,0.093384,-0.014618,-0.129150,-0.238770,-0.085327,0.051300,-0.020004,0.010063,-0.084351,-0.003567,0.064941,-0.205322,-0.158936,-0.074768,0.104370,0.197021,-0.080688,-0.066772,-0.036346,0.034912,-0.019760,0.110474,0.128662,0.094727,0.024948,-0.033356,-0.081848,0.054474,-0.065857,-0.156494,0.002527,0.097595,-0.027420,0.039185,0.063965,0.220093,0.029556,-0.115417]", "sports, camera, live streaming, Huawei, peripherals", "[0.017029,0.008209,-0.037292,0.218262,0.066895,0.142944,-0.032501,-0.015961,-0.097229,-0.095398,-0.065186,-0.048401,0.000182,-0.118896,-0.036743,0.041534,-0.122742,0.118591,0.140869,0.009567,-0.059875,0.007156,-0.098999,-0.044861,0.000140,0.096985,-0.149658,0.002346,0.018112,0.052338,-0.046265,0.119690,-0.048737,0.001713,0.072937,-0.079346,-0.012535,0.032623,0.026627,0.034973,0.053528,-0.022110,0.026733,0.060089,-0.116516,-0.043518,-0.043488,-0.044159,0.157715,-0.030762,-0.019791,-0.052338,0.197876,-0.002966,-0.009422,-0.065125,0.061096,-0.001552,0.113220,-0.085083,0.112976,-0.037415,-0.020096,-0.104492,-0.078430,0.051971,-0.040314,0.006413,0.120361,-0.072876,-0.048584,-0.091797,0.079041,0.094971,0.072144,0.093323,-0.007469,-0.046417,0.023453,0.113220,-0.029449,0.015457,0.018341,0.076111,-0.041443,-0.023071,0.064941,-0.119324,0.003418,-0.183472,0.017746,0.037506,-0.092468,-0.247925,0.035217,0.042206,-0.043060,0.049866,-0.071533,0.055115,0.087463,-0.107849,-0.105774,-0.088013,0.138794,0.133667,-0.105591,0.080017,-0.291260,0.065918,-0.010307,0.137695,0.077942,0.031372,-0.031494,-0.066772,-0.184082,0.121826,0.027359,-0.025604,-0.125244,0.035828,0.082153,0.065979,0.053650,0.204834,0.141479,-0.093140]", "[0.038727,0.080627,0.012520,-0.097778,-0.079651,-0.080627,0.072510,0.233887,0.063354,0.089355,-0.100342,0.022797,0.071655,-0.029770,0.036591,0.004120,-0.107788,0.074402,0.158081,-0.136841,0.073792,0.012650,-0.062500,-0.005177,-0.011795,0.089233,0.091309,-0.053497,0.007099,-0.111084,-0.009789,0.107178,0.033752,-0.034943,-0.006947,-0.010880,-0.069763,-0.074768,0.031769,-0.041870,0.011497,-0.126343,0.004051,-0.104675,0.011032,-0.054047,0.055023,0.161987,-0.164551,0.123413,-0.059692,0.093262,-0.070251,-0.083435,0.007645,0.035187,0.016373,-0.101807,-0.049683,0.023773,-0.117249,0.076172,0.039886,0.032227,0.119385,-0.098328,-0.024857,0.098328,-0.148804,0.002064,-0.116150,-0.015526,0.060181,-0.213379,0.043121,-0.017761,-0.075012,-0.008507,-0.011986,0.002188,0.042938,0.152100,0.112793,0.095215,-0.020905,-0.202515,0.006176,-0.072998,0.022263,-0.116516,-0.092773,0.024002,0.086182,-0.016418,-0.125366,0.116821,-0.047943,0.076355,-0.056732,-0.284424,-0.076416,-0.120667,-0.178833,-0.066956,0.080627,0.037872,0.024857,0.038666,0.021423,-0.125854,-0.032867,0.062256,0.017380,0.040985,-0.127441,0.170898,-0.062408,-0.010910,-0.122925,0.041534,-0.036530,-0.108398,0.146973,0.110962,-0.067322,0.070740,0.013237,0.000191]", "[0.012939,-0.063782,0.031616,0.127441,-0.055023,0.004395,-0.048828,0.045105,-0.079712,-0.024704,0.035126,-0.028763,0.033447,-0.102478,0.095581,0.066284,-0.001293,0.021500,0.074463,0.107361,-0.078918,-0.048767,-0.079773,-0.009651,0.115417,-0.087463,-0.055420,-0.059265,-0.019897,-0.050842,-0.187256,0.060608,-0.195801,-0.069092,0.148193,-0.093994,0.113037,0.062378,-0.014488,-0.116638,0.049225,-0.089050,0.122498,-0.005947,0.012444,-0.003235,-0.075439,0.046814,0.160522,-0.087769,0.039917,-0.022247,0.178955,0.099670,-0.016327,-0.021088,0.061279,0.109436,0.073547,0.035980,0.016708,0.017685,-0.053894,-0.045624,-0.031647,-0.041107,-0.079407,-0.067383,-0.015472,-0.054199,-0.014114,-0.164185,0.099182,-0.027191,-0.021957,-0.014603,0.093750,-0.001885,0.074951,0.094238,-0.001523,0.099731,0.045288,0.104187,-0.105957,0.030701,0.108154,-0.116394,0.090820,-0.153809,0.016983,0.059662,-0.109070,-0.269043,-0.109253,0.053131,-0.002453,0.025360,-0.108093,-0.054871,0.079834,-0.245972,-0.136841,-0.063293,0.174927,0.122498,-0.104980,-0.130249,-0.051605,0.057556,-0.071106,0.078369,0.081848,0.039459,0.021927,-0.000520,-0.001396,0.064575,-0.087463,-0.095398,-0.015175,0.164062,-0.089600,-0.004246,0.054199,0.225464,0.050018,-0.049957]", 1234567890123456781, 3.1411, true, "12345678");';
  await rdbStoreVector.execute(sqlInsertVector1);
  // ...
}

async insertInvIdxDB() {
  let sqlInsert1 : string =
    'INSERT INTO files (fileid, filename, keywords, chapter, abstract, content) VALUES ("0", "Sports live streaming scenario", "Sports, camera, live streaming, Huawei, peripherals", "", "Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario Sports live streaming scenario", "dummy");'
  await rdbStoreInvIdx.execute(sqlInsert1);
  // ...
}

Configure the information and context objects related to the inverted index and vector databases, and generate the retriever object for subsequent retrieval.

let globalRetriever:retrieval.Retriever | undefined;

async getRetriever() {
  let vectorDBConfig:retrieval.DbConfig = {
    name:'vector_test.db',
    securityLevel:relationalStore.SecurityLevel.S3
  }

  let invidxDBConfig:retrieval.DbConfig = {
    name:'files_test.db',
    securityLevel:relationalStore.SecurityLevel.S3
  }
  let context: Context | undefined = this.getUIContext().getHostContext();
  if (context == undefined) {
    console.info('getHostContext failed.');
    return;
  }
  let channelConfigVector:retrieval.ChannelConfig = {
    channelType:retrieval.ChannelType.VECTOR_DATABASE,
    context:context,
    dbConfig:vectorDBConfig
  }

  let channelConfigInvIdx:retrieval.ChannelConfig = {
    channelType:retrieval.ChannelType.INVERTED_INDEX_DATABASE,
    context:context,
    dbConfig:invidxDBConfig
  }

  let retrievalConfig:retrieval.RetrievalConfig = {
    channelConfigs:[channelConfigInvIdx, channelConfigVector]
  }

  await retrieval.getRetriever(retrievalConfig)
    .then((retriever:retrieval.Retriever) => {
      globalRetriever = retriever;
      console.info('GetRetriever success');
    })
    .catch((err:BusinessError) => {
      globalRetriever = undefined;
      console.error('Failure in getRetriever and code is ' + err.code);
    })
}

Use the retriever obtained in the previous step to perform retrieval based on the retrieval conditions.

let fieldWeight:Record<string, number> = {
  'filename':4.0
}

let fieldSlops:Record<string, number> = {
  'filename':5
}

let bm25Strategy:retrieval.Bm25Strategy = {
  bm25Weight:1.5,
  columnWeight:fieldWeight
}

let exactStrategy:retrieval.ExactMatchingStrategy = {
  exactMatchingWeight:1.2,
  columnWeight:fieldWeight
}

let outOfOrderStrategy:retrieval.ProximityStrategy = {
  proximityWeight:1.0,
  columnWeight:fieldWeight,
  columnSlops:fieldSlops
}

let invertedIndexStrategies:Array<retrieval.InvertedIndexStrategy> = [bm25Strategy, exactStrategy, outOfOrderStrategy]

let recallConditionInvIdx:retrieval.InvertedIndexRecallCondition ={
  ftsTableName:'files',
  primaryKey:['fileid'],
  fromClause:'files',
  responseColumns:['fileid', 'filename', 'keywords'],
  deepSize:2,
  invertedIndexStrategies:invertedIndexStrategies,
  recallName:'invIdxRecall'
}

// Modify the query representation vector specified for floatArray based on the service requirements.
let floatArray = new Float32Array([0.006954, -0.079041, 0.046173, 0.157959, -0.017212, 0.037018, -0.072083, -0.028488, -0.099854, 0.044037, -0.008911, -0.063049, 0.035950, -0.105835, 0.057739, 0.060364, -0.062042, 0.044159, 0.143188, 0.123901, -0.069641, -0.061920, -0.086731, -0.092468, 0.092957, -0.027649, -0.005497, -0.039276, 0.017502, -0.046570, -0.115906, 0.081177, -0.153931, -0.040588, 0.123474, -0.099060, 0.062042, 0.026352, -0.041382, -0.099548, 0.071167, -0.120850, 0.082642, 0.026398, -0.035614, -0.008545, -0.076660, -0.031067, 0.192017, -0.052582, 0.005310, 0.052734, 0.199463, 0.075195, -0.070740, -0.035950, 0.073120, 0.089172, 0.075989, 0.003582, 0.050201, -0.012787, 0.016647, -0.053619, 0.001906, -0.060181, -0.068359, -0.114502, -0.045013, 0.004547, -0.004673, -0.148071, 0.126343, 0.019394, -0.063110, -0.055908, 0.071228, 0.002369, 0.041412, 0.126709, -0.053467, 0.127808, 0.055420, 0.206177, 0.002169, -0.001452, 0.095520, -0.042511, 0.099243, -0.164185, 0.093384, -0.014618, -0.129150, -0.238770, -0.085327, 0.051300, -0.020004, 0.010063, -0.084351, -0.003567, 0.064941, -0.205322, -0.158936, -0.074768, 0.104370, 0.197021, -0.080688, -0.066772, -0.036346, 0.034912, -0.019760, 0.110474, 0.128662, 0.094727, 0.024948, -0.033356, -0.081848, 0.054474, -0.065857, -0.156494, 0.002527, 0.097595, -0.027420, 0.039185, 0.063965, 0.220093, 0.029556, -0.115417]);

let vectorQuery:retrieval.VectorQuery = {
  column:'keywords',
  value:floatArray,
  similarityThreshold:0.35
}

let recallConditionVector:retrieval.VectorRecallCondition = {
  vectorQuery:vectorQuery,
  fromClause:'vector',
  primaryKey:['fileid'],
  responseColumns:['filename_text', 'filename', 'int64_value', 'double_value', 'bool_value', 'blob_value'],
  recallName:'vectorRecall',
  deepSize:2
}

let vectorWeights:Record<string, number> = {
  'vectorRecall':0.5
}

let invidxWeights:Record<string, number> = {
  'vectorRecall':0.5
}

let vectorRerankParameter:retrieval.VectorRerankParameter = {
  vectorWeights:vectorWeights,
  thresholds:[0.55, 0.45]
}
let invidxRerankParameter:retrieval.InvertedIndexRerankParameter = {
  invertedIndexWeights:invidxWeights,
}

let parameters:Record<retrieval.ChannelType, retrieval.RerankParameter> = {
  0:vectorRerankParameter,
  1:invidxRerankParameter
}

let rerankMethod:retrieval.RerankMethod = {
  rerankType:retrieval.RerankType.RRF,
  parameters:parameters,
  isSoftmaxNormalized:true
}

let groundTruthIds: Array<string> = ['1','2', '3'];
let explain : retrieval.ExplanationConfig ={
  groundTruths: groundTruthIds
}

let retrievalCondition:retrieval.RetrievalCondition = {
  rerankMethod:rerankMethod,
  recallConditions:[recallConditionInvIdx, recallConditionVector],
  resultCount:2,
  explanation:explain
}

async retrieveRdb() {
  if (globalRetriever != undefined) {
    let query:string = 'Sports live streaming scenario'
    // Perform retrieval.
    globalRetriever.retrieveRdb(query, retrievalCondition)
      .then((rdbdata:retrieval.RdbRecords) => {

        // ...
        console.info('RetrieveRdb success.');
      })
      .catch((err:BusinessError) => {
        console.error('Failure in retrieveRdb and code is ' + err.code);
      })
  }
}

目录

Intelligent Data Retrieval-ArkTS

Use Cases