Help with adapting RenderTarget->ReadPixels() to a multithreaded framework

Development Programming & Scripting
1

I am trying to access pixel data and save images from an in-game camera to disk. Initially, the simple approach was to use a render target and subsequently RenderTarget->ReadPixels(), but as the native implementation of ReadPixels() contains a call to FlushRenderingCommands(), it would block the game thread until the image is saved. Being a computationally intensive operation, this was lowering my FPS way too much.

To solve this problem, I am trying to create a dedicated thread that can access the camera as a CaptureComponent, and then follow a similar approach. But as the FlushRenderingCommands() block can only be called from a game thread, I had to rewrite ReadPixels() without that call, (in a non-blocking way of sorts, inspired by the tutorial at A new, community-hosted Unreal Engine Wiki - Announcements - Unreal Engine Forums): but even then I am facing a problem with my in-game FPS being jerky whenever an image is saved (I confirmed this is not because of the actual saving to disk operation, but because of the pixel data access). My rewritten ReadPixels() function looks as below, I was hoping to get some suggestions as to what could be going wrong here. I am not sure if ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER can be called from a non-game thread, and if that’s part of my problem!

APIPCamera* cam = GameThread->pawn1->getFpvCamera();
	USceneCaptureComponent2D* capture = cam->getCaptureComponent(EPIPCameraType::PIP_CAMERA_TYPE_SCENE, true);
	if (capture != nullptr) {
		if (capture->TextureTarget != nullptr) {
			FTextureRenderTargetResource* RenderResource = capture->TextureTarget->GetRenderTargetResource();
			if (RenderResource != nullptr) {
				width = capture->TextureTarget->GetSurfaceWidth();
				height = capture->TextureTarget->GetSurfaceHeight();
				// Read the render target surface data back.	
				struct FReadSurfaceContext
				{
					FRenderTarget* SrcRenderTarget;
					TArray<FColor>* OutData;
					FIntRect Rect;
					FReadSurfaceDataFlags Flags;
				};

				bmp.Reset();
				FReadSurfaceContext ReadSurfaceContext =
				{
					RenderResource,
					&bmp,
					FIntRect(0, 0, RenderResource->GetSizeXY().X, RenderResource->GetSizeXY().Y),
					FReadSurfaceDataFlags(RCM_UNorm, CubeFace_MAX)
				};

				ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER(
					ReadSurfaceCommand,
					FReadSurfaceContext, Context, ReadSurfaceContext,
					{
						RHICmdList.ReadSurfaceData(
							Context.SrcRenderTarget->GetRenderTargetTexture(),
							Context.Rect,
							*Context.OutData,
							Context.Flags
						);
					});
			}
		}
	}
2

I’m facing the same problem. Have you found a solution for the problem?

3

Hmmm - I’m not sure if this was solved by changes to the engine since 4.15, but I’ve had the same essential problem and came up with similar code. In my case, the ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER macro worked fine - it was just a matter of waiting for it to finish. This thread solution solved my FPS drop when taking a picture from an in-game camera. In our case, this was a real issue, because it’s a VR app, and dropping FPS brought you temporarily into the steam VR loading screen. All fixed now.

bool FSavePhotoTask::ThreadSafe_ReadPixels(FRenderTarget* RT, TArray< FColor >& OutImageData, FReadSurfaceDataFlags InFlags, FIntRect InRect)
{
	if (InRect == FIntRect(0, 0, 0, 0))
	{
		InRect = FIntRect(0, 0, RT->GetSizeXY().X, RT->GetSizeXY().Y);
	}

	// Read the render target surface data back.	
	struct FReadSurfaceContext
	{
		FRenderTarget* SrcRenderTarget;
		TArray<FColor>* OutData;
		FIntRect Rect;
		FReadSurfaceDataFlags Flags;
	};
	OutImageData.Reset();
	FReadSurfaceContext ReadSurfaceContext =
	{
		RT,
		&OutImageData,
		InRect,
		InFlags
	};

	ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER(
		ReadSurfaceCommand,
		FReadSurfaceContext, Context, ReadSurfaceContext,
		{
			RHICmdList.ReadSurfaceData(
				Context.SrcRenderTarget->GetRenderTargetTexture(),
				Context.Rect,
				*Context.OutData,
				Context.Flags
			);
		});
	while (OutImageData.Num() == 0)
	{
		FWindowsPlatformProcess::Sleep(1.0f);
	}
	// A final second to be sure
	FWindowsPlatformProcess::Sleep(1.0f);

	return OutImageData.Num() > 0;
}
4

Hi,

How did you do that ?

When i’m trying to add the macro it fails to compile.

Bests,

Alex

5

Hello Alex,
you have to adjust your “projectname.build.cs” file.

mine looks like this

// Copyright 1998-2018 Epic Games, Inc. All Rights Reserved.

using UnrealBuildTool;

public class projectname : ModuleRules
{
public projectname(ReadOnlyTargetRules Target) : base(Target)
{
PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
bEnableExceptions = true;

	PublicDependencyModuleNames.AddRange(new string[] 
	{ 
		"Core", 
		"CoreUObject", 
		"Engine", 
		"InputCore", 
		"PhysXVehicles", 
		"HeadMountedDisplay", 
		"AIModule", 
		"GameplayTasks",
		"Json",
		"ImageWrapper",
		"RenderCore", 
		"RHI",
		"PhysXVehicleLib",
		"PhysX",
		"APEX"
	});

    PrivateDependencyModuleNames.AddRange(new string[] 
	{ 
		"UMG", 
		"Slate", 
		"SlateCore" 
	});
	
	PublicDefinitions.Add("HMD_MODULE_INCLUDED=1");
}

}

6

Hi Markus,

Thanks a lot :slight_smile: I could make it worked. It’s especially RHI module which seems to handle that.

Bests