Text Recognition

Service Introduction

The text recognition service can extract text from images of receipts, business cards, and documents. This service is useful for industries such as printing, education, and logistics. You can use it to create apps that handle data entry and check tasks.

This service can run on the cloud or device, but the supported languages differ in the two scenarios. The on-device API can recognize text in Simplified Chinese, Japanese, Korean, and Latin-based languages (including English, Spanish, Portuguese, Italian, German, French, Russian, and special characters. For details about the supported special characters, please refer to Latin Script Supported by On-device Text Recognition). The on-cloud API supports more languages such as Simplified Chinese, English, Spanish, Portuguese, Italian, German, French, Russian, Japanese, Korean, Polish, Finnish, Norwegian, Swedish, Danish, Turkish, Thai, Arabic, and Hindi.

The text recognition service is able to recognize text in both static images and dynamic camera streams with a host of APIs, which you can call synchronously or asynchronously to build your text recognition-enabled apps.

Use Cases

People may exchange their business cards in social networking, technical communications, business meetings, and many other scenarios. The text recognition service quickly recognizes key information in business cards and records them into the desired system. In the express field, this service detects images to recognize their contained information such as the recipient name, phone number, and address, and fills the information into the required places. It helps users get out of the dilemma of manually inputting text, making your apps more attractive.

Precautions

The text recognition service running on the cloud recognizes text with higher accuracy than that running on the device. However, the on-cloud one is unable to support text recognition from camera streams and cannot work in synchronous mode. If you want to use the on-cloud service of text recognition, enable ML Kit first. The text analyzer is created by different methods for text recognition from images on the device in synchronous mode, text recognition from images on the device in asynchronous mode, text recognition from images on the cloud in asynchronous mode, and text recognition from camera streams on the device. For details about how to create the text analyzer, please refer to the sample code in the development procedure.

Development Procedure

Before app development, you need to make necessary development preparations, configure the Maven repository address for the HMS Core SDK, and integrate the text recognition SDK.

Text Recognition from Images on the Device

1. Create the text analyzer MLTextAnalyzer to recognize text in images. You can set MLLocalTextSetting to specify languages that can be recognized. If you do not set the languages, only Latin-based languages can be recognized by default.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   // Method 1: Use default parameter settings to configure the on-device text analyzer. Only Latin-based languages can be recognized.
   MLTextAnalyzer analyzer = MLAnalyzerFactory.getInstance().getLocalTextAnalyzer();
   
   // Method 2: Use the customized parameter MLLocalTextSetting to configure the text analyzer on the device.
   MLLocalTextSetting setting = new MLLocalTextSetting.Factory()
       .setOCRMode(MLLocalTextSetting.OCR_DETECT_MODE)
       // Specify languages that can be recognized.
       .setLanguage("zh")
       .create();
   MLTextAnalyzer analyzer = MLAnalyzerFactory.getInstance().getLocalTextAnalyzer(setting);

   Collapse

   Word wrap

   Dark theme

   Copy code

   // Method 1: Use default parameter settings to configure the on-device text analyzer. Only Latin-based languages can be recognized.
    val analyzer = MLAnalyzerFactory.getInstance().localTextAnalyzer
   
   // Method 2: Use the customized parameter MLLocalTextSetting to configure the text analyzer on the device.
   val setting = MLLocalTextSetting.Factory()
            .setOCRMode(MLLocalTextSetting.OCR_DETECT_MODE) // Set languages that can be recognized.
            .setLanguage("zh")
            .create()
    val analyzer = MLAnalyzerFactory.getInstance().getLocalTextAnalyzer(setting)

2. Create an MLFrame object using android.graphics.Bitmap. JPG, JPEG, PNG, and BMP images are supported. It is recommended that the length-width ratio range from 1:2 to 2:1.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   // Create an MLFrame object using the bitmap, which is the image data in bitmap format.
   MLFrame frame = MLFrame.fromBitmap(bitmap);

   Collapse

   Word wrap

   Dark theme

   Copy code

   // Create an MLFrame object using the bitmap, which is the image data in bitmap format.
    val frame = MLFrame.fromBitmap(bitmap)

3. Pass the MLFrame object to the asyncAnalyseFrame method for text recognition.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   Task<MLText> task = analyzer.asyncAnalyseFrame(frame);
   task.addOnSuccessListener(new OnSuccessListener<MLText>() {
       @Override
       public void onSuccess(MLText text) {
         // Processing for successful recognition.
       }
   }).addOnFailureListener(new OnFailureListener() {
       @Override
       public void onFailure(Exception e) {
           // Processing logic for recognition failure.
       }
   });

   Collapse

   Word wrap

   Dark theme

   Copy code

   val task = analyzer.asyncAnalyseFrame(frame)
   task.addOnSuccessListener {
        // Processing for successful recognition.
   }.addOnFailureListener {
        // Processing logic for recognition failure.
   }

   The sample code uses the asynchronous call mode. Local text recognition also supports synchronous call using analyseFrame. The recognition result is specified by the MLText.Block array.

   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   Context context = getApplicationContext();
   MLTextAnalyzer analyzer = new MLTextAnalyzer.Factory(context).setLocalOCRMode(MLLocalTextSetting.OCR_DETECT_MODE).setLanguage("zh").create();
   SparseArray<MLText.Block> blocks = analyzer.analyseFrame(frame);

   Collapse

   Word wrap

   Dark theme

   Copy code

   val context = applicationContext
   val analyzer = MLTextAnalyzer.Factory(context).setLocalOCRMode(MLLocalTextSetting.OCR_DETECT_MODE).setLanguage("zh").create()
   val blocks = analyzer!!.analyseFrame(frame)

4. After the recognition is complete, stop the analyzer to release recognition resources.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   try {
       if (analyzer != null) {
           analyzer.stop(); 
       }
   } catch (IOException e) {
       // Exception handling.
   }

   Collapse

   Word wrap

   Dark theme

   Copy code

   try {
        if (analyzer != null) {
            analyzer.stop()
        }
    } catch (e: IOException) {
        // Exception handling.
    }

Text Recognition from Images on the Cloud

1. Set authentication information for your app. For details, please refer to Notes on Using Cloud Authentication Information.
2. Create an analyzer. The recommended way is to using MLRemoteTextSetting. In this way, you can specify to-be-recognized languages for more accurate text recognition.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   // Method 1: Use customized parameter settings.
   // Create a language set.
   List<String> languageList = new ArrayList();
   languageList.add("zh");
   languageList.add("en");
   // Set parameters.
   MLRemoteTextSetting setting = new MLRemoteTextSetting.Factory()
       // Set the font type for the on-cloud API of text recognition.
       // When handwriting is set, only sparse text is supported as the detection mode, only Chinese (zh) is supported in the language list, and only the coordinates of four corner points of the quadrilateral are supported as the format of the returned text border box.
       // Configurations set via methods like setTextDensityScene, setLnaguageList, and setBorderType do not take effect.
       // When printscript is set, manually configure the language list, detection mode, and format of the returned text border box. If not, their default configurations will be adopted automatically.
       // MLRemoteTextSetting.OCR_HANDWRITTENFONT_SCENE: handwriting.
       // MLRemoteTextSetting.OCR_PRINTFONT_SCENE: printscript.
       .setTextFontScene(MLRemoteTextSetting.OCR_HANDWRITTENFONT_SCENE)
       // Set the on-cloud text detection mode.
       // MLRemoteTextSetting.OCR_COMPACT_SCENE: dense text recognition.
       // MLRemoteTextSetting.OCR_LOOSE_SCENE: sparse text recognition.
       .setTextDensityScene(MLRemoteTextSetting.OCR_LOOSE_SCENE)
       // Specify the languages that can be recognized, which should comply with ISO 639-1.
       .setLanguageList(languageList)
       // Set the format of the returned text border box.
       // MLRemoteTextSetting.NGON: Return the coordinates of the four corner points of the quadrilateral.
       // MLRemoteTextSetting.ARC: Return the corner points of a polygon border in an arc. The coordinates of up to 72 corner points can be returned.
       .setBorderType(MLRemoteTextSetting.ARC)
       .create();
   MLTextAnalyzer analyzer = MLAnalyzerFactory.getInstance().getRemoteTextAnalyzer(setting);
   // Method 2: Use the default parameter settings to automatically detect languages for text recognition. This method is applicable to sparse text scenarios. The format of the returned text box is MLRemoteTextSetting.NGON.
   MLTextAnalyzer analyzer = MLAnalyzerFactory.getInstance().getRemoteTextAnalyzer();

   Collapse

   Word wrap

   Dark theme

   Copy code

   // Method 1: Use customized parameter settings.
   // Create a language set.
   val languageList: MutableList<String?> = ArrayList< String?>()
   languageList.add("zh")
   languageList.add("en")
   // Set parameters.
   val setting = MLRemoteTextSetting.Factory() // Set the on-cloud text detection mode:
            // MLRemoteTextSetting.OCR_COMPACT_SCENE: dense text recognition
            // MLRemoteTextSetting.OCR_LOOSE_SCENE: sparse text recognition
            .setTextDensityScene(MLRemoteTextSetting.OCR_LOOSE_SCENE) // Specify the languages that can be recognized, which should comply with ISO 639-1.
            .setLanguageList(languageList) // Set the format of the returned text border box.
            // MLRemoteTextSetting.NGON: Return the coordinates of the four corner points of the quadrilateral.
       // MLRemoteTextSetting.ARC: Return the corner points of a polygon border in an arc. The coordinates of up to 72 corner points can be returned.
            .setBorderType(MLRemoteTextSetting.ARC)
            .create()
   val analyzer = MLAnalyzerFactory.getInstance().getRemoteTextAnalyzer(setting)
   // Method 2: Use the default parameter settings to automatically detect languages for text recognition. This method is applicable to sparse text scenarios. The format of the returned text box is MLRemoteTextSetting.NGON.
   val analyzer = MLAnalyzerFactory.getInstance().remoteTextAnalyzer

3. Create an MLFrame object using android.graphics.Bitmap. JPG, JPEG, PNG, and BMP images are supported.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   MLFrame frame = MLFrame.fromBitmap(bitmap);

   Collapse

   Word wrap

   Dark theme

   Copy code

   val frame = MLFrame.fromBitmap(bitmap)

4. Pass the created MLFrame object to the asyncAnalyseFrame method of the analyzer for text recognition. For details about the result codes, please refer to MLException.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   Task<MLText> task = analyzer.asyncAnalyseFrame(frame);
   task.addOnSuccessListener(new OnSuccessListener<MLText>() {
       @Override
       public void onSuccess(MLText text) {
           // Recognition success.
       }
   }).addOnFailureListener(new OnFailureListener() {
       @Override
       public void onFailure(Exception e) {
           // If the recognition fails, obtain related exception information.
           try {        
               MLException mlException = (MLException) e;
               // Obtain the result code. You can process the result code and customize respective messages displayed to users.
               int errorCode = mlException.getErrCode();
              // Obtain the error information. You can quickly locate the fault based on the result code.
               String errorMessage = mlException.getMessage();    
           } catch (Exception error) {
              // Handle the conversion error.
           }
       }
   });

   Collapse

   Word wrap

   Dark theme

   Copy code

   val task = analyzer!!.asyncAnalyseFrame(frame)
   task.addOnSuccessListener {
        // Recognition success.
   }.addOnFailureListener { e ->
        // If the recognition fails, obtain related exception information.
        try {
            val mlException = e as MLException
            // Obtain the result code. You can process the result code and customize respective messages displayed to users.
            val errorCode = mlException.errCode
            // Obtain the error information. You can quickly locate the fault based on the result code.
            val errorMessage = mlException.message
        } catch (error: Exception) {
           // Handle the conversion error.
        }
    }

5. After the recognition is complete, stop the analyzer to release recognition resources.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   if (analyzer != null) {  
       try {
           analyzer.stop();
       } catch (IOException e) {
       // Exception handling.
       }
   }

   Collapse

   Word wrap

   Dark theme

   Copy code

   if (analyzer != null) {
        try {
            analyzer.stop()
        } catch (e: IOException) {
           // Exception handling.
        }
    }

Text Recognition from Camera Streams on the Device

Your app can process camera streams, convert camera frames into the MLFrame object, and recognize text using the local static image recognition method. If the synchronous recognition API is called, your app can also use the LensEngine class built in the SDK to locally detect text in camera streams and create and initialize a LensEngine object. For details, please refer to the sample code:

1. Create an analyzer.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   // Method 1: Use default parameter settings to configure the text analyzer. Only Latin-based languages can be recognized.
   MLTextAnalyzer analyzer = new MLTextAnalyzer.Factory(context).create();
   
   // Method 2: Use the custom parameter MLTextAnalyzer.Factory to configure the text analyzer. Other supported languages can be recognized.
   MLTextAnalyzer.Factory factory = new MLTextAnalyzer.Factory(context);
   // Specify languages that can be recognized.
   factory.setLanguage("zh");
   MLTextAnalyzer analyzer = factory.create();

   Collapse

   Word wrap

   Dark theme

   Copy code

   // Method 1: Use default parameter settings to configure the text analyzer. Only Latin-based languages can be recognized.
   val analyzer = MLTextAnalyzer.Factory(context).create()
   
   // Method 2: Use the custom parameter MLTextAnalyzer.Factory to configure the text analyzer. Other supported languages can be recognized.
   val factory = MLTextAnalyzer.Factory(context)
   // Specify languages that can be recognized.
   factory.setLanguage("zh")
   val analyzer = factory.create()

2. Create the OcrDetectorProcessor class for processing recognition results. This class implements the MLAnalyzer.MLTransactor<T> API and uses the transactResult method in this class to obtain the recognition results and implement specific services.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   public class OcrDetectorProcessor implements MLAnalyzer.MLTransactor<MLText.Block> {
       @Override
       public void transactResult(MLAnalyzer.Result<MLText.Block> results) {
           SparseArray<MLText.Block> items = results.getAnalyseList();
           // Determine detection result processing as required. Note that only the detection results are processed.
           // Other detection-related APIs provided by ML Kit cannot be called.
       }
       @Override
       public void destroy() {
           // Callback method used to release resources when the detection ends.
       }
   }

   Collapse

   Word wrap

   Dark theme

   Copy code

   class OcrDetectorProcessor : MLTransactor<MLText.Block?> {
        override fun transactResult(results: MLAnalyzer.Result<MLText.Block?>) {
            val items = results.analyseList
            // Determine detection result processing as required. Note that only the detection results are processed.
           // Other detection-related APIs provided by ML Kit cannot be called.
        }
   
        override fun destroy() {
           // Callback method used to release resources when the detection ends.
        }
    }

3. Set the detection result processor to bind the analyzer to the result processor.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   analyzer.setTransactor(new OcrDetectorProcessor());

   Collapse

   Word wrap

   Dark theme

   Copy code

   analyzer!!.setTransactor(OcrDetectorProcessor())

4. Call the synchronous API to use the built-in LensEngine of the SDK to create an object, register the analyzer, and initialize camera parameters.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   LensEngine lensEngine = new LensEngine.Creator(getApplicationContext(),analyzer)
       .setLensType(LensEngine.BACK_LENS)
       .applyDisplayDimension(1440, 1080)
       .applyFps(30.0f)
       .enableAutomaticFocus(true)
       .create();

   Collapse

   Word wrap

   Dark theme

   Copy code

   val lensEngine = LensEngine.Creator(applicationContext, analyzer)
            .setLensType(LensEngine.BACK_LENS)
            .applyDisplayDimension(1440, 1080)
            .applyFps(30.0f)
            .enableAutomaticFocus(true)
            .create()

5. Call the run method to launch the camera, read camera streams, and perform recognition. (For details, please refer to the sample code to implement other logic of the SurfaceView control.)
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   SurfaceView mSurfaceView = findViewById(R.id.surface_view);
   try {
       lensEngine.run(mSurfaceView.getHolder());
   } catch (IOException e) {
       // Exception handling logic.
   }

   Collapse

   Word wrap

   Dark theme

   Copy code

   val mSurfaceView: SurfaceView? = findViewById(R.id.surface_view);
    try {
        lensEngine!!.run(mSurfaceView!!.holder)
    } catch (e: IOException) {
        // Exception handling logic.
    }

6. After the detection is complete, stop the analyzer to release detection resources.
   Java

   Kotlin

   Collapse

   Word wrap

   Dark theme

   Copy code

   if (analyzer != null) {
       try {
           analyzer.stop();
       } catch (IOException e) {
       // Exception handling.
       }
   }
   if (lensEngine != null) {
       lensEngine.release();
   }

   Collapse

   Word wrap

   Dark theme

   Copy code

   if (analyzer != null) {
        try {
            analyzer.stop()
        } catch (e: IOException) {
           // Exception handling.
        }
    }
   lensEngine?.release()

In camera stream detection, when MLAnalyzer.MLTransactor<T> is inherited to process detection results, if your app needs to stop detection after a specific result is detected and continue detection after the result is processed, please refer to Development for Multi Detections in Camera Stream Detection Mode.