We use essential cookies for the website to function, as well as analytics cookies for analyzing and creating statistics of the website performance. To agree to the use of analytics cookies, click "Accept All". You can manage your preferences at any time by clicking "Cookie Settings" on the footer. More Information.

Only Essential Cookies
Accept All
Best PracticesIndustry SolutionsSocialAI-Enhanced Image and Text Creation

AI-Enhanced Image and Text Creation

Overview

This solution is tailored for image and text creation in social communication applications, leveraging HarmonyOS features like seamless hopping and interactive services.

Introduction to the Scenario

The process starts with selecting local images via Photo Picker, followed by intelligent image processing. Users can also capture videos using a custom camera. During text creation, the workflow supports seamless continuation of editing and cross-device access to Gallery or camera content. Users can seamlessly continue editing and photo access across devices.

Demo

The process of editing images and texts is as follows.

Solution Overview

Usage Scope

This solution is ideal for social communication applications, outlining how to integrate HarmonyOS features for image and text creation. It offers a detailed technical implementation guide to accelerate application development.

Advantages

This solution combines HarmonyOS features such as device interconnection, intelligent processing, and hopping to improve user experience when posting contents online. The detailed advantages are as follows:

(1) Device interconnection: allows users to flexibly select media resources from various devices and take photos using different devices, simplifying the process of data transmission while providing a more convenient experience.

(2) HarmonyOS AI: provides powerful support for creation. It enables users to extract information from images for text composition and remove backgrounds to extract objects for secondary creation, working wonders on editing.

(3) Hopping: enables seamless work across devices and synchronizes the latest editing status to the new device, allowing users to continue their work on the most suitable device.

Scenario Analysis

Typical Scenarios

Scenario

Description

Implementation

Image selection

Select a resource file type on the release page

Photo Picker for image selection

Photo shoot

Take and preview moving photos on the custom camera page.

Camera component for custom camera

Image text recognition, cropping, and HDR Vivid image display

Select an image on the image browsing page to crop objects, and copy the texts on the image for text content creation. Automatically identify the HDR mode and display highlights.

Image component for OCR and cropping

Cross-device photo selection

Select an image from another device and send it back to the local device.

CollaborationService component

Hopping for the editing page

Continue editing on different devices.

Hopping capability for continuous editing across devices via ArkData and distributed file management

Implementation

Photo Picker

Scenario Description

When a user initiates the release process from the home page, a semi-modal Picker page is displayed, which can be customized to provide more options.

Key Points

The system Picker does not require the READ_IMAGEVIDEO and WRITE_IMAGEVIDEO permissions.

Core Code

Create a PhotoViewPicker instance first.

Collapse
Word wrap
Dark theme
Copy code
  1. const photoViewPicker = new photoAccessHelper.PhotoViewPicker();

Set the attributes of image selection based on the service logic, for example, set the media resource type and the maximum number of resources.

Collapse
Word wrap
Dark theme
Copy code
  1. const photoSelectOptions = new photoAccessHelper.PhotoSelectOptions();
  2. photoSelectOptions.MIMEType = photoAccessHelper.PhotoViewMIMETypes.IMAGE_TYPE;
  3. photoSelectOptions.maxSelectNumber = CommonConstants.LIMIT_PICKER_NUM - selectedNum;
Obtain images.
Collapse
Word wrap
Dark theme
Copy code
  1. photoViewPicker.select(photoSelectOptions).then((photoSelectResult: photoAccessHelper.PhotoSelectResult) => {
  2. let uriArr = photoSelectResult.photoUris;
  3. callback(uriArr);
  4. }).catch((err: BusinessError) => {
  5. Logger.error(UIUtils.tag,
  6. `Invoke photoViewPicker.select failed, code is ${err.code}, message is ${err.message}`);
  7. });

OCR, AI Cropping, and HDR Vivid

Scenario Description

After selecting an image, users can browse it, press and hold an object to extract it, or recognize text for further editing. If the image is taken in HDR Vivid mode, the enhanced visual effect is displayed.

Demo

Long press the image for text recognition and object cropping

Key Points

(1) Set the enableAnalyzer attribute of the Image component to implement Optical character Recognition (OCR) and cropping. Set the dynamicRangeMode attribute to enable HDR, which must be used together with image.DecodingOptions to configure the dynamic range mode.

(2) OCR: If text can be recognized from an image, users can tap the recognition button on the image or long press the text to display a shortcut menu for copying or selecting text.

(3) Cropping: Long press an object in the image to crop it. You can copy and share the cropped image from the menu.

Core Code

Enable AI image analysis and set the dynamic range mode of the image.

Collapse
Word wrap
Dark theme
Copy code
  1. Image(item)
  2. .objectFit(ImageFit.Contain)
  3. .enableAnalyzer(true)
  4. .dynamicRangeMode(DynamicRangeMode.HIGH)

Set image decoding options, which is used together with the dynamic range mode.

Collapse
Word wrap
Dark theme
Copy code
  1. public static options: image.DecodingOptions = {
  2. index: 0,
  3. editable: false,
  4. desiredPixelFormat: image.PixelMapFormat.RGBA_8888,
  5. };
  6. static createPixelMap(uri: string): ImageInfo | undefined {
  7. let imageInfo: ImageInfo | undefined;
  8. try {
  9. let file = fileIo.openSync(uri, fileIo.OpenMode.READ_ONLY);
  10. let displayName = file.name;
  11. let imageResource = image.createImageSource(file.fd);
  12. let pixelMap = imageResource.createPixelMapSync(FileUtils.options);
  13. imageInfo = { imagePixelMap: pixelMap, imageName: displayName };
  14. fileIo.closeSync(file);
  15. } catch (error) {
  16. Logger.error(FileUtils.tag, `createPixelMap error: ${JSON.stringify(error)}`);
  17. }
  18. return imageInfo;
  19. }

Moving Photo

Scenario Description

On the image editing page, add a custom camera tab to display diverse photos taken in different modes.

Key Points

(1) After the camera is initialized, users can enable/disable the moving photo feature (disabled by default). If this feature is enabled, you can tap the Moving Photo button to switch the status.

(2) A photo is displayed 30 seconds after it is taken.

(3) MovingPhotoView is required for previewing a photo; users can long press the photo to play it.

(4) Apply for necessary permissions. The camera can be initialized after the permissions are granted.

Core Code

Apply for the required permissions.

Collapse
Word wrap
Dark theme
Copy code
  1. private permissions: Array<Permissions> = [
  2. 'ohos.permission.CAMERA',
  3. 'ohos.permission.MICROPHONE',
  4. 'ohos.permission.MEDIA_LOCATION',
  5. 'ohos.permission.READ_IMAGEVIDEO',
  6. 'ohos.permission.WRITE_IMAGEVIDEO',
  7. ];
  8. abilityAccessCtrl.createAtManager().requestPermissionsFromUser(DataUtils.context, this.permissions).then(() => {
  9. this.surfaceId = this.mXComponentController.getXComponentSurfaceId();
  10. this.initCamera();
  11. this.getThumbnail();
  12. })

Set the Moving Photo attribute of the camera.

Collapse
Word wrap
Dark theme
Copy code
  1. setEnableLivePhoto(isMovingPhoto: boolean) {
  2. try {
  3. if (this.photoOutput?.isMovingPhotoSupported()) {
  4. this.photoOutput?.enableMovingPhoto(isMovingPhoto);
  5. }
  6. } catch (error) {
  7. Logger.error(this.tag, `The setEnableLivePhoto call failed. error: ${JSON.stringify(error)}`);
  8. }
  9. }
Get the address and thumbnail of the latest image in the media library.
Collapse
Word wrap
Dark theme
Copy code
  1. async getThumbnail(): Promise<void> {
  2. try {
  3. let photoAsset: photoAccessHelper.PhotoAsset =
  4. AppStorage.get(CommonConstants.KEY_PHOTO_ASSET) as photoAccessHelper.PhotoAsset;
  5. if (photoAsset === undefined) {
  6. return;
  7. }
  8. this.currentImg = await photoAsset.getThumbnail();
  9. } catch (error) {
  10. Logger.error(this.tag, `getThumbnail error: ${JSON.stringify(error)}`);
  11. }
  12. }

Import the Moving Photo library.

Collapse
Word wrap
Dark theme
Copy code
  1. import { MovingPhotoView, MovingPhotoViewController, MovingPhotoViewAttribute } from '@ohos.multimedia.movingphotoview';

Request the moving photo feature through photoAccessHelper.PhotoAsset.

Collapse
Word wrap
Dark theme
Copy code
  1. @StorageLink(CommonConstants.KEY_MOVING_DATA) src: photoAccessHelper.MovingPhoto | undefined = undefined;
  2. @StorageLink(CommonConstants.KEY_IMAGE_INFO) imageInfoArr: Array<ImageInfo> = [];
  3. @State isMuted: boolean = false;
  4. async aboutToAppear(): Promise<void> {
  5. // ...
  6. this.requestMovingPhoto();
  7. }
  8. private requestMovingPhoto() {
  9. let photoAsset: photoAccessHelper.PhotoAsset =
  10. AppStorage.get(CommonConstants.KEY_PHOTO_ASSET) as photoAccessHelper.PhotoAsset;
  11. if (photoAsset === undefined) {
  12. return;
  13. }
  14. let requestOptions: photoAccessHelper.RequestOptions = {
  15. deliveryMode: photoAccessHelper.DeliveryMode.FAST_MODE,
  16. }
  17. photoAccessHelper.MediaAssetManager.requestMovingPhoto(DataUtils.context, photoAsset, requestOptions,
  18. new MediaDataHandlerMovingPhoto()).catch(() => {
  19. Logger.error(this.tag, `requestMovingPhoto fail!`);
  20. });
  21. }
  22. class MediaDataHandlerMovingPhoto implements photoAccessHelper.MediaAssetDataHandler<photoAccessHelper.MovingPhoto> {
  23. async onDataPrepared(movingPhoto: photoAccessHelper.MovingPhoto): Promise<void> {
  24. AppStorage.setOrCreate(CommonConstants.KEY_MOVING_DATA, movingPhoto);
  25. }
  26. }

Add a moving photo display.

Collapse
Word wrap
Dark theme
Copy code
  1. build() {
  2. Flex({
  3. direction: new BreakpointType(
  4. {
  5. sm: FlexDirection.Column,
  6. md: FlexDirection.Column,
  7. lg: FlexDirection.Row,
  8. }
  9. ).getValue(this.currentBreakpoint),
  10. wrap: FlexWrap.NoWrap,
  11. justifyContent: FlexAlign.Start,
  12. alignItems: ItemAlign.Start,
  13. alignContent: FlexAlign.Start
  14. }) {
  15. this.setActions();
  16. MovingPhotoView({
  17. movingPhoto: this.src,
  18. controller: this.controller
  19. })
  20. .width($r('app.string.full_screen'))
  21. .objectFit(ImageFit.Contain)
  22. .muted(this.isMuted)
  23. .margin(new BreakpointType(
  24. {
  25. sm: { bottom: $r('app.float.margin_190') } as Padding,
  26. md: { bottom: $r('app.float.margin_190') } as Padding,
  27. lg: { right: $r('app.float.margin_24') } as Padding,
  28. }
  29. ).getValue(this.currentBreakpoint))
  30. }
  31. .backgroundColor(Color.Black)
  32. .width($r('app.string.full_screen'))
  33. .height($r('app.string.full_screen'))
  34. }
NOTE

This section describes only the core code of the main process. Customizing a camera requires other configurations, for which you can refer to the CameraService file. To learn about how to use the moving photo feature, see the related API description @ohos.multimedia.movingphotoview (MovingPhotoView).

CollaborationService

Demo

Cross-device image capture

Scenario Description

The CollaborationService component enables cross-device access to albums, camera, and images, facilitating image transmission between devices.

Key Points

The CollaborationService component requires a network connection and the same login account.

NOTICE

Currently, the CollaborationService component allows only the heavy devices (not portable) to invoke light devices (portable). For example:

(1) The tablet can invoke a phone rather than another tablet.

(2) The phone cannot invoke a tablet or another phone.

Core Code

Taking photos and accessing albums across devices

Use createCollaborationServiceMenuItems to define the device list picker and display the list of devices with the camera capability in the network.

Collapse
Word wrap
Dark theme
Copy code
  1. import {
  2. CollaborationServiceFilter,
  3. CollaborationServiceStateDialog,
  4. createCollaborationServiceMenuItems
  5. } from '@kit.ServiceCollaborationKit';
  6. @Builder
  7. CollaborationMenu() {
  8. Menu() {
  9. createCollaborationServiceMenuItems([CollaborationServiceFilter.ALL]);
  10. }
  11. }

Use the CollaborationCameraStateDialog component to display a dialog box to show the camera status on the target device.

This component is directly called in the build() function to implement the onState method. After the shooting is complete, the content is returned through the onState method.

The callback function of the onState method contains two parameters: stateCode (which indicates the service completion status), and buffer (which indicates the data returned upon success).

Collapse
Word wrap
Dark theme
Copy code
  1. @Builder
  2. setCollaborationDialog() {
  3. CollaborationServiceStateDialog({
  4. onState: (stateCode: number, bufferType: string, buffer: ArrayBuffer): void => this.doInsertPicture(stateCode,
  5. bufferType, buffer)
  6. });
  7. }
  8. private doInsertPicture(stateCode: number, bufferType: string, buffer: ArrayBuffer): void {
  9. if (stateCode !== 0) {
  10. Logger.error(this.tag, `doInsertPicture stateCode: ${stateCode}}`);
  11. return;
  12. }
  13. Logger.info(this.tag, `doInsertPicture bufferType: ${bufferType}}`);
  14. if (bufferType === CommonConstants.BUFFER_TYPE) {
  15. if (this.imageInfoArr.length === CommonConstants.LIMIT_PICKER_NUM) {
  16. try {
  17. this.getUIContext().getPromptAction().showToast({
  18. message: $r('app.string.toast_picker_limit'),
  19. duration: DataUtils.fromResToNumber($r('app.float.show_DELAY_TIME')),
  20. });
  21. } catch (error) {
  22. Logger.error(this.tag, `showToast error: ${JSON.stringify(error)}}`);
  23. }
  24. return;
  25. }
  26. let saveUri: string = FileUtils.saveFile(DataUtils.context, buffer);
  27. let imageInfo: ImageInfo | undefined = FileUtils.createPixelMap(saveUri);
  28. if (imageInfo) {
  29. this.imageInfoArr.unshift(imageInfo);
  30. this.selectedData.unshiftData(imageInfo.imagePixelMap);
  31. // copy file to distributedFilesDir
  32. FileUtils.copyFileToDestination(saveUri, DataUtils.context.distributedFilesDir);
  33. }
  34. }
  35. }

Application Continuation

Scenario Description

The hopping capability allows users to continue editing on other devices.

Demo

Editing continuation enabled

Key Points

Prerequisites:

(1) The same HUAWEI ID is logged on two devices.

(2) Both devices enable Wi-Fi and Bluetooth and connect to the same LAN for faster data transmission.

(3) Application continuation can be triggered only within the same application (UIAbility), and the application must be available on both devices.

(4) The data transmitted by using wantParam in the onContinue callback must be less than 100 KB. For a large amount of data, such as images, the distributed data objects or distributed file system is required.

To request permissions, set the requestPermissions attribute of the module object in the module.json5 file.

Collapse
Word wrap
Dark theme
Copy code
  1. "requestPermissions": [
  2. {
  3. "name": "ohos.permission.DISTRIBUTED_DATASYNC",
  4. "reason": "$string:distributed_desc",
  5. "usedScene": {
  6. "abilities": [
  7. "EntryAbility"
  8. ],
  9. "when": "always"
  10. }
  11. }
  12. // ...
  13. ]
Set continuable in the abilities field of the module object in the module.json5 file to true.
Collapse
Word wrap
Dark theme
Copy code
  1. "abilities": [
  2. {
  3. // ...
  4. "continuable": true,
  5. // ...
  6. }
  7. ]

Core Code

You can migrate the route stack on demand or dynamically configure the route stack to enable continuation only for specific pages. The following sample code uses GraphicCreationPage as an example. For details about how to migrate the route stack on demand, see Migrating the Page Stack on Demand.

Collapse
Word wrap
Dark theme
Copy code
  1. onPageShow(): void {
  2. DataUtils.context.setMissionContinueState(AbilityConstant.ContinueState.ACTIVE, (result) => {
  3. Logger.info('setMissionContinueState ACTIVE result: ', `${result.code}`);
  4. });
  5. }
  6. aboutToDisappear(): void {
  7. this.title = '';
  8. this.description = '';
  9. DataUtils.context.setMissionContinueState(AbilityConstant.ContinueState.INACTIVE, (result) => {
  10. Logger.info('setMissionContinueState INACTIVE result: ', `${result.code}`);
  11. });
  12. }

Specify the page to enable continuation.

Collapse
Word wrap
Dark theme
Copy code
  1. onWindowStageRestore(windowStage: window.WindowStage) {
  2. windowStage.loadContent('pages/GraphicCreationPage', (err, data) => {
  3. // ...
  4. });
  5. }
Implement the onContinue API on the migration side, use the asset card array to pass images seamlessly across devices, and package the images with other text data into a data object.
Collapse
Word wrap
Dark theme
Copy code
  1. async onContinue(wantParam: Record<string, Object | undefined>): Promise<AbilityConstant.OnContinueResult> {
  2. try {
  3. // get distribute id
  4. let sessionId: string = distributedDataObject.genSessionId();
  5. wantParam.distributedSessionId = sessionId;
  6. // set images assets info
  7. let imageInfoArray = AppStorage.get<Array<ImageInfo>>(CommonConstants.KEY_IMAGE_INFO);
  8. let assets: commonType.Assets = [];
  9. if (imageInfoArray) {
  10. for (let i = 0; i < imageInfoArray.length; i++) {
  11. let append = imageInfoArray[i];
  12. let attachment: commonType.Asset | undefined = this.getAssetInfo(append);
  13. if (attachment === undefined) {
  14. continue;
  15. }
  16. assets.push(attachment);
  17. }
  18. }
  19. // set distribute data object
  20. let contentInfo: ContentInfo = new ContentInfo(
  21. AppStorage.get(CommonConstants.KEY_TITLE),
  22. AppStorage.get(CommonConstants.KEY_DESCRIPTION),
  23. AppStorage.get(CommonConstants.KEY_IMAGE_INFO),
  24. assets
  25. );
  26. let source = contentInfo.flatAssets();
  27. // save data to distribute
  28. this.distributedObject = distributedDataObject.create(this.context, source);
  29. Logger.info(this.tag, `onContinue source: ${JSON.stringify(source)}`);
  30. this.distributedObject.setSessionId(sessionId);
  31. await this.distributedObject.save(wantParam.targetDevice as string).catch((err: BusinessError) => {
  32. Logger.error(this.tag, `Failed to save. Code: ${err.code}, message: ${err.message}`);
  33. });
  34. } catch (error) {
  35. Logger.error(this.tag, 'distributedDataObject failed', `code ${(error as BusinessError).code}`);
  36. }
  37. return AbilityConstant.OnContinueResult.AGREE;
  38. }
  39. private getAssetInfo(append: ImageInfo): commonType.Asset | undefined {
  40. let filePath = this.context.distributedFilesDir + '/' + append.imageName;
  41. try {
  42. fileIo.statSync(filePath);
  43. let uri: string = fileUri.getUriFromPath(filePath);
  44. let stat = fileIo.statSync(filePath);
  45. let attachment: commonType.Asset = {
  46. name: append.imageName,
  47. uri: uri,
  48. path: filePath,
  49. createTime: stat.ctime.toString(),
  50. modifyTime: stat.ctime.toString(),
  51. size: stat.size.toString()
  52. };
  53. Logger.info(this.tag, `getAssetInfo attachment = ${JSON.stringify(attachment)}`);
  54. return attachment;
  55. } catch (error) {
  56. Logger.error(this.tag, `getAssetInfo error: ${JSON.stringify(error)}`);
  57. return undefined;
  58. }
  59. }
NOTE

The code is written in EntryAbility instead of Page. AppStorage can be used to continuously store the data and bind the data bidirectionally. When the data changes, the view is changed.

Implement the onCreate and onNewWant APIs on the target device. The onCreate API is called during cold start or during hot start in multiton mode. The onNewWant API is called during hot start in singleton mode. Register a data listener and restore the data of page transition only when the application is in the continuation state.

Collapse
Word wrap
Dark theme
Copy code
  1. onCreate(want: Want, launchParam: AbilityConstant.LaunchParam): void {
  2. DataUtils.context = this.context;
  3. // set circulation status INACTIVE
  4. this.context.setMissionContinueState(AbilityConstant.ContinueState.INACTIVE, (result) => {
  5. Logger.info(`restoreDistributedObject setMissionContinueState code: ${result.code}`);
  6. });
  7. this.restoreDistributedObject(want, launchParam);
  8. Logger.info(this.tag, '%{public}s', 'Ability onCreate');
  9. }
  10. onNewWant(want: Want, launchParam: AbilityConstant.LaunchParam): void {
  11. this.restoreDistributedObject(want, launchParam);
  12. }
  13. private restoreDistributedObject(want: Want, launchParam: AbilityConstant.LaunchParam): void {
  14. if (launchParam.launchReason !== AbilityConstant.LaunchReason.CONTINUATION) {
  15. return;
  16. }
  17. try {
  18. // File copying takes a long time, resulting in the page lifecycle aboutToAppear being executed first.
  19. let imageInfoArr: ImageInfo[] = [];
  20. AppStorage.setOrCreate(CommonConstants.KEY_IMAGE_INFO, imageInfoArr);
  21. let contentInfo: ContentInfo = new ContentInfo(undefined, undefined, undefined, undefined);
  22. // Create a distributed data object.
  23. this.distributedObject = distributedDataObject.create(this.context, contentInfo);
  24. // Add a data restored listener.
  25. this.distributedObject.on('status',
  26. (_sessionId: string, _networkId: string, status: 'online' | 'offline' | 'restored') => {
  27. if (status === 'restored') {
  28. if (!this.distributedObject) {
  29. return;
  30. }
  31. AppStorage.setOrCreate(CommonConstants.KEY_TITLE, this.distributedObject['title']);
  32. AppStorage.setOrCreate(CommonConstants.KEY_DESCRIPTION, this.distributedObject['description']);
  33. let attachments = this.distributedObject['attachments'] as commonType.Assets;
  34. if (attachments) {
  35. for (const attachment of attachments) {
  36. let sourceUri: string =
  37. fileUri.getUriFromPath(`${this.context.distributedFilesDir}/${attachment.name}`);
  38. let destination: string = this.context.filesDir;
  39. FileUtils.copyFileToDestination(sourceUri, destination);
  40. let uri: string = `${this.context.filesDir}/${attachment.name}`;
  41. let imageInfo = FileUtils.createPixelMap(uri);
  42. if (imageInfo) {
  43. imageInfoArr.push(imageInfo);
  44. }
  45. }
  46. }
  47. AppStorage.set(CommonConstants.KEY_IMAGE_INFO, imageInfoArr);
  48. AppStorage.setOrCreate(CommonConstants.KEY_RESTORE_IMAGE_INFO, imageInfoArr);
  49. }
  50. });
  51. let sessionId: string = want.parameters?.distributedSessionId as string;
  52. this.distributedObject.setSessionId(sessionId);
  53. this.context.restoreWindowStage(new LocalStorage());
  54. } catch (error) {
  55. Logger.info(`restoreDistributedObject error: ${JSON.stringify(error)}`);
  56. }
  57. }

The image transfer requires the support from the distributed file system. After the sender saves the file to the distribution directory, the receiver can copy this file from the distribution directory to the local sandbox.

Collapse
Word wrap
Dark theme
Copy code
  1. static copyFileToDestination(sourceUri: string, destination: string) {
  2. try {
  3. let buf = new ArrayBuffer(CommonConstants.FILE_BUFFER_SIZE);
  4. let readSize = 0;
  5. let file = fileIo.openSync(sourceUri, fileIo.OpenMode.READ_ONLY);
  6. let readLen = fileIo.readSync(file.fd, buf, { offset: readSize });
  7. let destinationDistribute =
  8. fileIo.openSync(`${destination}/${file.name}`, fileIo.OpenMode.READ_WRITE | fileIo.OpenMode.CREATE);
  9. while (readLen > 0) {
  10. readSize += readLen;
  11. fileIo.writeSync(destinationDistribute.fd, buf);
  12. readLen = fileIo.readSync(file.fd, buf, { offset: readSize });
  13. }
  14. Logger.info(FileUtils.tag, 'copyFileToDestination success');
  15. fileIo.closeSync(file);
  16. fileIo.closeSync(destinationDistribute);
  17. } catch (err) {
  18. Logger.error(FileUtils.tag, `copyFileToDestination failed. Code: ${err.code}, message: ${err.message}`);
  19. }
  20. }
Search in Best Practices
Enter a keyword.