预览帧处理时如何选择format

问题现象

在双路预览开发时，可能会出现下面两种异常场景：

画面色彩混乱、画面错位。 画面出现大面积偏色或者局部色彩错误，甚至出现色块杂乱等情况，如下图所示（图一为正常图案，图二三为异常现象）：

预览界面直接闪退。 相机预览界面打开后快速闪退，并伴有如下crash日志：

Process name:com.example.personaltest
Process life time:4s
Process Memory(kB): 242962(Rss)
Device Memory(kB): Total 16017036, Free 771396, Available 7143424
Reason:Signal:SIGSEGV(SEGV_MAPERR)@0x0000005b5f0be000 
Fault thread info:
Tid:54261, Name:OS_FFRT_2_2
#00 pc 000000000001d8b8 /system/lib64/libhispeed_image.so(HSD_Image_RGBA_to_rgbA+120)(3eaf5c382d6322f2adbc21b9e86197ea)
#01 pc 0000000000fc9024 /system/lib64/libskia_canvaskit.z.so(SkConvertPixels(SkImageInfo const&, void*, unsigned long, SkImageInfo const&, void const*, unsigned long)+356)(7098d0b1d31420779f28556b877e2b29)
#02 pc 0000000000fddddc /system/lib64/libskia_canvaskit.z.so(SkBitmap::writePixels(SkPixmap const&, int, int)+344)(7098d0b1d31420779f28556b877e2b29)
#03 pc 00000000000043a0 /system/lib64/platformsdk/libpixelconvertadapter.z.so(OHOS::Media::PixelConvertAdapter::WritePixelsConvert(void const*, unsigned int, OHOS::Media::ImageInfo const&, void*, OHOS::Media::Position const&, unsigned int, OHOS::Media::ImageInfo const&) (.cfi)+664)(5368e33cc90cfc45cc93c5145375d9db)
#04 pc 000000000010d8b4 /system/lib64/platformsdk/libimage_native.z.so(OHOS::Media::PixelConvert::PixelsConvert(OHOS::Media::BufferInfo const&, OHOS::Media::BufferInfo&, int, bool) (.cfi)+7220)(cbd62a0ff3d78c60f180e38726228131)
#05 pc 00000000000f8d14 /system/lib64/platformsdk/libimage_native.z.so(OHOS::Media::PixelMap::Create(unsigned int const*, unsigned int, OHOS::Media::BuildParam&, OHOS::Media::InitializationOptions const&, int&) (.cfi)+748)(cbd62a0ff3d78c60f180e38726228131)
#06 pc 00000000000f89e8 /system/lib64/platformsdk/libimage_native.z.so(OHOS::Media::PixelMap::Create(unsigned int const*, unsigned int, OHOS::Media::InitializationOptions const&) (.cfi)+208)(cbd62a0ff3d78c60f180e38726228131)
#07 pc 00000000000d14b4 /system/lib64/platformsdk/libimage_napi.z.so(OHOS::Media::CreatePixelMapExec(napi_env__*, void*) (.cfi)+464)(8a4937c6722b56ca6a98bec4c504c48f)
#08 pc 000000000005d130 /system/lib64/platformsdk/libace_napi.z.so(NativeAsyncWork::AsyncWorkCallback(uv_work_s*)+304)(0cebf964e89dab5a2ce4cefd8e58beea)
#09 pc 00000000000135bc /system/lib64/platformsdk/libuv.so(uv__queue_work+60)(00cee7c8797de4a98569c4744411c891)
#10 pc 00000000000ad964 /system/lib64/ndk/libffrt.so(ffrt::UVTask::ExecuteImpl(ffrt::UVTask*, void (*)(ffrt_executor_task*, int))+256)(5c382a50e711fcf0b33083e1da8d8ed1)
#11 pc 00000000000a8f4c /system/lib64/ndk/libffrt.so(ffrt::ExecuteTask(ffrt::TaskBase*)+252)(5c382a50e711fcf0b33083e1da8d8ed1)
#12 pc 000000000006abe4 /system/lib64/ndk/libffrt.so(ffrt::CPUWorker::RunTask(ffrt::TaskBase*, ffrt::CPUWorker*)+84)(5c382a50e711fcf0b33083e1da8d8ed1)
#13 pc 000000000006ae88 /system/lib64/ndk/libffrt.so(ffrt::CPUWorker::WorkerLooper(ffrt::CPUWorker*)+396)(5c382a50e711fcf0b33083e1da8d8ed1)
#14 pc 00000000000474d0 /system/lib64/ndk/libffrt.so(ffrt::CPUWorker::Dispatch(ffrt::CPUWorker*)+212)(5c382a50e711fcf0b33083e1da8d8ed1)
#15 pc 00000000000472a4 /system/lib64/ndk/libffrt.so(ffrt::CPUWorker::WrapDispatch(void*)+60)(5c382a50e711fcf0b33083e1da8d8ed1)
#16 pc 00000000001d8938 /system/lib/ld-musl-aarch64.so.1(start+240)(73c8ccfd08a34c45800c16fb998a68cc)
========SubmitterStacktrace========
#00 pc 000000000001382c /system/lib64/platformsdk/libuv.so(uv_queue_work_internal+316)(00cee7c8797de4a98569c4744411c891)
#01 pc 000000000005ce6c /system/lib64/platformsdk/libace_napi.z.so(NativeAsyncWork::Queue(NativeEngine*)+412)(0cebf964e89dab5a2ce4cefd8e58beea)
#02 pc 000000000008bf38 /system/lib64/platformsdk/libace_napi.z.so(napi_queue_async_work+40)(0cebf964e89dab5a2ce4cefd8e58beea)
#03 pc 00000000000c0488 /system/lib64/platformsdk/libimage_napi.z.so(OHOS::Media::PixelMapNapi::CreatePixelMap(napi_env__*, napi_callback_info__*) (.cfi)+524)(8a4937c6722b56ca6a98bec4c504c48f)
#04 pc 0000000000066fd4 /system/lib64/platformsdk/libace_napi.z.so(panda::JSValueRef ArkNativeFunctionCallBack<true>(panda::JsiRuntimeCallInfo*)+292)(0cebf964e89dab5a2ce4cefd8e58beea)
#05 pc 0000000000e0d9f0 /system/lib64/module/arkcompiler/stub.an(RTStub_PushCallArgsAndDispatchNative+44)
#06 pc 0000000000465018 /system/lib64/module/arkcompiler/stub.an(BCStub_HandleCallthis2Imm8V8V8V8StwCopy+436)
#07 at anonymous entry (entry/src/main/ets/pages/Index.ets:230:38)
#08 pc 00000000004a60b0 /system/lib64/platformsdk/libark_jsruntime.so(panda::ecmascript::InterpreterAssembly::Execute(panda::ecmascript::EcmaRuntimeCallInfo*)+608)(3beab3d3995f1b1c11f542938d599a63)
#09 pc 00000000004a5644 /system/lib64/platformsdk/libark_jsruntime.so(panda::FunctionRef::CallForNapi(panda::ecmascript::EcmaVM const*, panda::JSValueRef*, panda::JSValueRef* const*, int)+536)(3beab3d3995f1b1c11f542938d599a63)
#10 pc 000000000007a7e4 /system/lib64/platformsdk/libace_napi.z.so(napi_call_function+212)(0cebf964e89dab5a2ce4cefd8e58beea)
#11 pc 0000000000093740 /system/lib64/platformsdk/libimage_napi.z.so(OHOS::Media::CommonCallbackRoutine(napi_env__*, OHOS::Media::ImageAsyncContext*&, napi_value__* const&)+444)(8a4937c6722b56ca6a98bec4c504c48f)
#12 pc 0000000000093a24 /system/lib64/platformsdk/libimage_napi.z.so(OHOS::Media::JsGetComponentCallBack(napi_env__*, napi_status, OHOS::Media::ImageAsyncContext*) (.cfi)+336)(8a4937c6722b56ca6a98bec4c504c48f)
#13 pc 000000000003f4f4 /system/lib64/platformsdk/libace_napi.z.so(NativeAsyncWork::AsyncAfterWorkCallback(uv_work_s*, int)+468)(0cebf964e89dab5a2ce4cefd8e58beea)
#14 pc 00000000000722c0 /system/lib64/platformsdk/libruntime.z.so(std::__h::__function::__func<OHOS::AbilityRuntime::OHOSJsEnvironmentImpl::PostTaskToHandler(uv_task_info_s const*)::$_0, std::__h::allocator<OHOS::AbilityRuntime::OHOSJsEnvironmentImpl::PostTaskToHandler(uv_task_info_s const*)::$_0>, void ()>::operator()()+128)(6fac88a3788b167d65c694c7b98b5186)
#15 pc 000000000001efe4 /system/lib64/chipset-sdk-sp/libeventhandler.z.so(OHOS::AppExecFwk::(anonymous namespace)::EventRunnerImpl::ExecuteEventHandler(std::__h::unique_ptr<OHOS::AppExecFwk::InnerEvent, void (*)(OHOS::AppExecFwk::InnerEvent*)>&)+1824)(cb50d8e3b11e95df8dfdd059d8351e8d)

碰到上述两种情况，应该如何正确处理？

背景知识

CameraFormat：相机输出流的原始数据格式，定义相机硬件输出图像的编码方式（如YUV、RGB等）。决定Surface接收数据的底层结构。
nextImage.format：ImageReceiver接收图像的格式，是相机输出流数据经系统封装后的中间格式，与CameraFormat强关联。
PixelMapFormat：PixelMap的处理格式，用于内存中处理图像数据的像素排列方式（如RGBA、NV21等）。

三者关系：CameraFormat决定nextImage.format，二次预览时则需要根据nextImage.format取值来选择对应的PixelMapFormat和单位像素字节数。芯片根据创建预览输出流时的CameraFormat类型来出图，因此nextImage.format是由CameraFormat来决定的。而在将nextImage处理成pixelMap时，需要根据nextImage.format的值选择对应的PixelMapFormat以及单位像素字节数。三者的对应关系如下表：

CameraFormat值	对应nextImage.format值	PixelMapFormat取值	单位像素字节数	适用的输出流
CAMERA_FORMAT_RGBA_8888	-	-	-	-
CAMERA_FORMAT_YUV_420_SP	25	image.PixelMapFormat.NV21	1.5	预览流，视频流（NORMAL_VIDEO）
CAMERA_FORMAT_JPEG	-	-	-	拍照流（NORMAL_PHOTO）
CAMERA_FORMAT_YCBCR_P010	35	image.PixelMapFormat.YCBCR_P010	3	预览流（NORMAL_VIDEO），视频流（NORMAL_VIDEO）
CAMERA_FORMAT_YCRCB_P010	36	image.PixelMapFormat.YCRCB_P010	3	预览流（NORMAL_VIDEO），视频流（NORMAL_VIDEO）
CAMERA_FORMAT_HEIC	-	-	-	-

注：表中的NORMAL_VIDEO指的是相机模式为NORMAL_VIDEO，NORMAL_PHOTO则是指相机模式为NORMAL_VIDEO，未特别标明的表示两种模式都可以。

问题定位

画面色彩杂乱。
画面色彩异常现象明显，根本原因在于像素格式解析错配，由于CameraFormat与PixelMapFormat取值不匹配，导致色彩空间不匹配或者色彩通道顺序错位，进而造成色彩错乱。
预览界面发生crash闪退。
1. 从日志可以看出crash是因为发生了严重的内存访问异常（SIGSEGV(SEGV_MAPERR)），从日志33行可以看出问题出在Index.ets:230:38，对应代码中操作为：
```
let pixelMapFormat = image.PixelMapFormat.YCRCB_P010;
pixelMap = await image.createPixelMap(dstArr.buffer, {
  size: { height: height, width: width },
  srcPixelFormat: pixelMapFormat,
});
```
2. image.createPixelMap方法是通过图像像素数据的缓冲区（dstArr.buffer）内容创建PixelMap，而dstArr.buffer的创建过程为：
```
let mSize = 1.5;
const dstBufferSize = width * height * mSize; // 以NV21为例（YUV_420_SP格式的图片）YUV_420_SP内存计算公式：长x宽+(长x宽)/2
const dstArr = new Uint8Array(dstBufferSize);
for (let j = 0; j < height * mSize; j++) {
  const srcBuf = new Uint8Array(imgComponent.byteBuffer, j * stride, width);
  dstArr.set(srcBuf, j * width);
}
```
3. 从上面两段代码可知，创建图像缓冲区数据时选择的单位像素字节数为1.5，而createPixelMap时传入的格式是PixelMapFormat.YCRCB_P010。PixelMapFormat.YCRCB_P010格式对应的单位像素字节数应该为3。

分析结论

画面色彩杂乱：画面出现错乱是因为在转成pixelMap处理时，选择错了pixelMapFormat，导致画面色彩混乱。
预览界面发生crash闪退：crash闪退是因为图像数组拷贝时，单位像素字节数选择过小，使得图像缓冲区数据长度不符合要求，导致创建pixelMap时发生地址越界造成闪退。

修改建议

参照背景知识章节中的表格，选择正确的相机模式和CameraFormat值，然后在处理nextImage时选择对应的PixelMapFormat和单位像素字节数。