On-device Text to Speech

Service Introduction

With the support of an on-device model, on-device text to speech (TTS) can convert text information into speech even when there is no Internet connection. The current version provides two types of male voice speakers and seven types of female voice speakers.

Timbres

Use Cases

This service is widely used in daily life. For example, in a voice navigation app, a specific navigation voice may be synthesized offline based on navigation data, to make navigation more personalized and allow users to enjoy the travel. This service can also be used for 24-hour broadcasting in railway stations, greatly reducing labor costs.

Precautions

• Currently, volume, speed, and pitch adjustment is supported only by the eagle model of on-device TTS.
• To use on-device TTS, you need to download an on-device model first.
• It is recommended that the text in a single request contain a maximum of 500 characters.
• Default specifications of the output audio data are as follows: PCM mono, 16-bit depth, and 16 kHz audio sampling rate.

Development Process

Before app development, you need to make necessary development preparations, configure the Maven repository address for the HMS Core SDK, and integrate the on-device text to speech SDK.

1. Set authentication information for your app. For details, please refer to Notes on Using Cloud Authentication Information.
2. Create a TTS engine. You can use the MLTtsConfig class to create a TTS engine. For details about the supported languages and speakers, please refer to MLTtsConstants.
   Java

   Kotlin

   // Use customized parameter settings to create a TTS engine.
   // For details about the speaker names, please refer to the Timbres section.
   MLTtsConfig mlTtsConfig = new MLTtsConfig()
       // Set the text converted from speech to English.
       .setLanguage(MLTtsConstants.TTS_EN_US)
       // Set the speaker with the English female voice (bee).
       .setPerson(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_FEMALE_BEE)
       // Set the TTS mode to on-device mode. The default mode is real-time mode.
       .setSynthesizeMode(MLTtsConstants.TTS_OFFLINE_MODE);
   MLTtsEngine mlTtsEngine = new MLTtsEngine(mlTtsConfig);
   // Update the configuration when the engine is running.
   mlTtsEngine.updateConfig(mlTtsConfig);

   // Use customized parameter settings to create a TTS engine.
   // For details about the speaker names, please refer to the Timbres section.
   var mlTtsConfig = MLTtsConfig() // Set the text converted from speech to English.
            .setLanguage(MLTtsConstants.TTS_EN_US) // Set the speaker with the English female voice (bee).
            .setPerson(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_FEMALE_BEE) // Set the TTS mode to on-device mode. The default mode is real-time mode.
            .setSynthesizeMode(MLTtsConstants.TTS_OFFLINE_MODE)
   val mlTtsEngine = MLTtsEngine(mlTtsConfig)
   // Update the configuration when the engine is running.
   mlTtsEngine.updateConfig(mlTtsConfig)

3. Create a TTS callback function to process the TTS result.
   Java

   Kotlin

   MLTtsCallback callback = new MLTtsCallback() {
       @Override
       public void onError(String taskId, MLTtsError err) {
           // Processing logic for TTS failure.
       }
   
       @Override
       public void onWarn(String taskId, MLTtsWarn warn) {
           // Alarm handling without affecting service logic.
       }
   
       @Override
           // Return the mapping between the currently played segment and text. start: start position of the audio segment in the input text; end (excluded): end position of the audio segment in the input text.
       public void onRangeStart(String taskId, int start, int end) {
           // Process the mapping between the currently played segment and text.
       }
   
       @Override
       // taskId: ID of a TTS task corresponding to the audio.
        // audioFragment: audio data.
       // offset: offset of the audio segment to be transmitted in the queue. One TTS task corresponds to a TTS queue.
       // range: text area where the audio segment to be transmitted is located; range.first (included): start position; range.second (excluded): end position.
       public void onAudioAvailable(String taskId, MLTtsAudioFragment audioFragment, int offset,
           Pair<Integer, Integer> range, Bundle bundle) {
           // Audio stream callback API, which is used to return the synthesized audio data to the app.
       }
   
       @Override
       public void onEvent(String taskId, int eventId, Bundle bundle) {
           // Callback method of a TTS event. eventId indicates the event name.
           boolean isInterrupted;
           switch (eventId) {
               case MLTtsConstants.EVENT_PLAY_START:
                   // Called when playback starts.
                   break;
               case MLTtsConstants.EVENT_PLAY_STOP:
                   // Called when playback stops.
                   isInterrupted = bundle.getBoolean(MLTtsConstants.EVENT_PLAY_STOP_INTERRUPTED);
                   break;
               case MLTtsConstants.EVENT_PLAY_RESUME:
                   // Called when playback resumes.
                   break;
               case MLTtsConstants.EVENT_PLAY_PAUSE:
                   // Called when playback pauses.
                   break;
               // Pay attention to the following callback events when you focus on only synthesized audio data but do not use the internal player for playback:
               case MLTtsConstants.EVENT_SYNTHESIS_START:
                   // Called when TTS starts.
                   break;
               case MLTtsConstants.EVENT_SYNTHESIS_END:
                   // Called when TTS ends.
                   break;
               case MLTtsConstants.EVENT_SYNTHESIS_COMPLETE:
                   // TTS is complete. All synthesized audio streams are passed to the app.
                   isInterrupted = bundle.getBoolean(MLTtsConstants.EVENT_SYNTHESIS_INTERRUPTED);
                   break;
               default:
                   break;
               }
          }
     };

   val callback: MLTtsCallback = object : MLTtsCallback {
        override fun onError(taskId: String, err: MLTtsError) {
            // Processing logic for TTS failure.
        }
   
        override fun onWarn(taskId: String, warn: MLTtsWarn) {
            // Alarm handling without affecting service logic.
        }
   
        // Return the mapping between the currently played segment and text. start: start position of the audio segment in the input text; end (excluded): end position of the audio segment in the input text.
        override fun onRangeStart(taskId: String, start: Int, end: Int) {
            // Process the mapping between the currently played segment and text.
        }
   
        // taskId: ID of an audio synthesis task corresponding to the audio.
        // audioFragment: audio data.
       // offset: offset of the audio segment to be transmitted in the queue. One TTS task corresponds to a TTS queue.
        // range: text area where the audio segment to be transmitted is located; range.first (included): start position; range.second (excluded): end position.
        override fun onAudioAvailable(taskId: String, audioFragment: MLTtsAudioFragment, offset: Int,
                                      range: Pair<Int, Int>, bundle: Bundle) {
            // Audio stream callback API, which is used to return the synthesized audio data to the app.
        }
   
        override fun onEvent(taskId: String, eventId: Int, bundle: Bundle) {
            // Callback method of a TTS event. eventId indicates the event name.
            val isInterrupted: Boolean
            when (eventId) {
                MLTtsConstants.EVENT_PLAY_START -> {
                }
                MLTtsConstants.EVENT_PLAY_STOP ->                         // Called when playback stops.
                    isInterrupted = bundle.getBoolean(MLTtsConstants.EVENT_PLAY_STOP_INTERRUPTED)
                MLTtsConstants.EVENT_PLAY_RESUME -> {
                }
                MLTtsConstants.EVENT_PLAY_PAUSE -> {
                }
                MLTtsConstants.EVENT_SYNTHESIS_START -> {
                }
                MLTtsConstants.EVENT_SYNTHESIS_END -> {
                }
                MLTtsConstants.EVENT_SYNTHESIS_COMPLETE ->                         // TTS is complete. All synthesized audio streams are passed to the app.
                    isInterrupted = bundle.getBoolean(MLTtsConstants.EVENT_SYNTHESIS_INTERRUPTED)
                else -> {
                }
            }
        }
    }

4. Pass the TTS callback to the TTS engine created in step 2 to perform TTS.
   Java

   Kotlin

   mlTtsEngine.setTtsCallback(callback);

   mlTtsEngine.setTtsCallback(callback)

5. Perform on-device TTS. To use the on-device TTS service, you need to download the model package of the corresponding speaker. You can use MLLocalModelManager to check whether the on-device model of the specified speaker has been downloaded.
   Java

   Kotlin

   // Create an on-device TTS model manager.
   MLLocalModelManager localModelManager = MLLocalModelManager.getInstance();
   // Create an MLTtsLocalModel instance to set the speaker so that the language model corresponding to the speaker can be downloaded through the model manager.
   MLTtsLocalModel model = new MLTtsLocalModel.Factory(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_FEMALE_BEE).create();
   localModelManager.isModelExist(model).addOnSuccessListener(new OnSuccessListener<Boolean>() {
       @Override
       public void onSuccess(Boolean aBoolean) {
           // If the model is not downloaded, call the download API. Otherwise, call the TTS API of the on-device engine.
           if (aBoolean) {
               // Call the speak API to perform TTS. sourceText indicates the text to be synthesized. For details about the speak API, please refer to step 6.
               mlTtsEngine.speak(sourceText, MLTtsEngine.QUEUE_APPEND | MLTtsEngine.OPEN_STREAM);
           } else {
               // Call the API for downloading the on-device TTS model in step 5.
               downloadModel(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_FEMALE_BEE);
           }
       }
   }).addOnFailureListener(new OnFailureListener() {
       @Override
       public void onFailure(Exception e) {
           // Query error message.
       }
   });

   // Create an on-device TTS model manager.
   val localModelManager = MLLocalModelManager.getInstance()
    // Create an MLTtsLocalModel instance to set the speaker so that the language model corresponding to the speaker can be downloaded through the model manager.
   val model = MLTtsLocalModel.Factory(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_FEMALE_BEE).create()
   localModelManager.isModelExist(model).addOnSuccessListener { aBoolean ->
        // If the model is not downloaded, call the download API. Otherwise, call the TTS API of the on-device engine.
        if (aBoolean) {
            // Call the speak API to perform TTS. sourceText indicates the text to be synthesized. For details about the speak API, please refer to step 6.
            mlTtsEngine.speak(sourceText, MLTtsEngine.QUEUE_APPEND or MLTtsEngine.OPEN_STREAM)
        } else {
            // Call the API for downloading the on-device TTS model in step 5.
            downloadModel(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_FEMALE_BEE)
        }
    }.addOnFailureListener {
        // Query error message.
   }

6. Use MLLocalModelManager to manually download the language model package required for on-device TTS.
   Java

   Kotlin

   private void downloadModel(String person) {
       // Create an on-device TTS model manager.
       MLLocalModelManager localModelManager = MLLocalModelManager.getInstance();
       // Create an MLTtsLocalModel instance and pass the speaker (indicating by person) to download the language model corresponding to the speaker.
       final MLTtsLocalModel model = new MLTtsLocalModel.Factory(person)
       .create();
       // Create a download policy configurator. You can set that when any of the following conditions is met, the model can be downloaded: 1. The device is charging; 2. Wi-Fi is connected; 3. The device is idle.
       MLModelDownloadStrategy request = new MLModelDownloadStrategy.Factory().needWifi().create();
       // Create a download progress listener for the on-device model.
       MLModelDownloadListener modelDownloadListener = new MLModelDownloadListener() {
           @Override
           public void onProcess(long alreadyDownLength, long totalLength) {
               // Display the download progress of the on-device language model.
           }
       };
       // Call the download API of the on-device model manager.
       localModelManager.downloadModel(model, request, modelDownloadListener)
           .addOnSuccessListener(new OnSuccessListener<Void>() {
               @Override
               public void onSuccess(Void aVoid) {
                   // Download success.
                   // Call updateConfig to update the engine model configuration.
                   mlTtsEngine.updateConfig(mlTtsConfig);
                   // Call the speak API to perform TTS. sourceText indicates the text to be synthesized. For details about the speak API, please refer to step 6.
                   mlTtsEngine.speak(sourceText, MLTtsEngine.QUEUE_APPEND | MLTtsEngine.OPEN_STREAM);
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(Exception e) {
                   // Download failure.
                }
            });
   }

   private fun downloadModel(person: String) {
        // Create an on-device TTS model manager.
        val localModelManager = MLLocalModelManager.getInstance()
        // Create an MLTtsLocalModel instance and pass the speaker (indicating by person) to download the language model corresponding to the speaker.
        val model = MLTtsLocalModel.Factory(person).create()
        // Create a download policy configurator. You can set that when any of the following conditions is met, the model can be downloaded: 1. The device is charging; 2. Wi-Fi is connected; 3. The device is idle.
        val request = MLModelDownloadStrategy.Factory().needWifi()
        .create()
        // Create a download progress listener for the on-device model.
        val modelDownloadListener = MLModelDownloadListener { alreadyDownLength, totalLength ->
            // Display the download progress of the on-device language model.
        }
        // Call the download API of the on-device model manager.
        localModelManager.downloadModel(model, request, modelDownloadListener)
                .addOnSuccessListener { // Download success.
                    // Call updateConfig to update the engine model configuration.
                    mlTtsEngine!!.updateConfig(mlTtsConfig)
                    // Call the speak API to perform TTS. sourceText indicates the text to be synthesized. For details about the speak API, please refer to step 6.
                    mlTtsEngine.speak(sourceText, MLTtsEngine.QUEUE_APPEND or MLTtsEngine.OPEN_STREAM)
                }
                .addOnFailureListener {
                    // Download failure.
                }
    }

7. Perform TTS. For details about how to use this API, please refer to speak.
   Java

   Kotlin

   // sourceText indicates the text to be synthesized.
   mlTtsEngine.speak(sourceText, MLTtsEngine.QUEUE_APPEND | MLTtsEngine.OPEN_STREAM);

   // sourceText indicates the text to be synthesized.
   mlTtsEngine!!.speak(sourceText, MLTtsEngine.QUEUE_APPEND or MLTtsEngine.OPEN_STREAM)

8. Control the playback (valid only when an internal player is used).
   Java

   Kotlin

   // Pause playback.
   mlTtsEngine.pause();
   // Resume playback.
   mlTtsEngine.resume();

   // Pause playback.
   mlTtsEngine.pause()
    // Resume playback.
    mlTtsEngine.resume()

9. Stop the ongoing TTS tasks and clear all TTS tasks in the queue. This method is valid for calling the internal player.
   Java

   Kotlin

   mlTtsEngine.stop();

   mlTtsEngine.stop()

10. Release resources after TTS ends.
    Java

    Kotlin

    if (mlTtsEngine != null) {
        mlTtsEngine.shutdown();
    }

    if (mlTtsEngine != null) {
         mlTtsEngine.shutdown()
     }