Accessing a RenderTarget's pixels on iOS fails

I have some code that works fine on PC but not iOS. I need to access the pixel data of a rendertarget to save as a jpg, this is mostly working barring locking the top mip’s pixel data.

When I use the render target as a source texture what I want to see gets rendered. When I force the pixel values of the generated UTexture2D to a pattern I see the pattern as expected but I don’t see valid data when I lock the top mip.

As mentioned this works on PC, but I need it to work on iOS too. If I have to I’ll do this all in ObjectiveC on a UIImage but that seems like an overkill and will be messy - plus make it harder to apply cool UE4 level materials.

In the code below RenderTexture is a UTextureRenderTarget2D. As far as I can tell it’s working perfectly.

static int iTest = -1;
UTexture2D *USharedRenderTarget::Create2DTexture() const
{
    //RenderTexture is valid, I can render it's contents but I'd like to access the data directly to save
    //as a jpg or png - right now I'm just trying to get it into a UTexture2D and render to verify.
    UTexture2D *p2dTex = UTexture2D::CreateTransient(RenderTexture->SizeX, RenderTexture->SizeY, RenderTexture->GetFormat());
#if WITH_EDITORONLY_DATA
    p2dTex->MipGenSettings = TMGS_NoMipmaps;
#endif
    p2dTex->SRGB = RenderTexture->SRGB;
    p2dTex->UpdateResource();               //Apply these settings
    
    
    // Read the pixels from the RenderTarget and store them in a FColor array
    TArray<FColor> SurfData;
    FRenderTarget *RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();
    RenderTarget->ReadPixels(SurfData);
    
    // Lock and copies the data between the textures
    auto* TextureData = (FColor*)p2dTex->PlatformData->Mips[0].BulkData.Lock(LOCK_READ_WRITE);
    
    //A few different tests
    if (iTest == 0)
    {
        //Ignoring TextureData
        //Does work - creates texture pattern expected
        for (int iY=0 ; iY<RenderTexture->SizeY ; iY++)
        {
            float fY = (float)iY / (float)(RenderTexture->SizeY-1);
            uint8 g = (uint8)(fY * 255.0f);
            uint8 a = 255;
            for (int iX=0 ; iX<RenderTexture->SizeX ; iX++)
            {
                auto &rCol = TextureData[iX + iY*RenderTexture->SizeX];
                float fX = (float)iX / (float)(RenderTexture->SizeX-1);
                
                rCol.R = (uint8)(fX * 255.0f);
                rCol.B = (uint8)((fX + fY) * 255.0f);
                rCol.A = a;
                rCol.G = g;
            }
        }
    }
    else if (iTest == 1)
    {
        //Does NOT work (actually it was just to remove the Memcpy as a test)
        FColor *pSrc =SurfData.GetData();
        FColor *pDest = TextureData;
        int iRemain = SurfData.Num();
        while (iRemain)
        {
            *pDest = *pSrc;
            --iRemain;
            ++pSrc;
            ++pDest;
        }
    }
    else if (iTest == 2)
    {
        //Does not compile - these Lock / Unlock Functions are not available outside the render code?
        /*
            uint32 SrcStride;
            uint32 DestStride;
            void* Src = RHILockTexture2D( RenderTexture, 0, RLM_ReadOnly, SrcStride, false );
            void* Dst = RHILockTexture2D( p2dTex, 0, RLM_WriteOnly, DestStride, false );
            
            check(SrcStride == DestStride);
            FMemory::Memcpy( Dst, Src, RenderTexture->SizeX * RenderTexture->SizeY * SrcStride);
            
            RHIUnlockTexture2D( RenderTexture, 0, false );
            RHIUnlockTexture2D( p2dTex, 0, false );
        */
    }
    else
    {
        //Does NOT work, original code
        const int32 TextureDataSize = SurfData.Num() * 4;
        FMemory::Memcpy(TextureData, SurfData.GetData(), TextureDataSize);
    }
    
    p2dTex->PlatformData->Mips[0].BulkData.Unlock();
    
    // Apply Texture changes to GPU memory
    p2dTex->UpdateResource();
    return p2dTex;
}

Setup code for reference:

USharedRenderTarget::USharedRenderTarget(const class FObjectInitializer& PCIP)
	: Super(PCIP)
{
	RenderTexture = PCIP.CreateDefaultSubobject<UTextureRenderTarget2D>(this, TEXT("RenderTexture"));
}

USharedRenderTarget::~USharedRenderTarget()
{
}

void USharedRenderTarget::Setup(int iWidth, int iHeight)
{
	m_iWidth = iWidth;
	m_iHeight = iHeight;
	RenderTexture->ClearColor = FLinearColor::Black;
	RenderTexture->InitAutoFormat(iWidth, iHeight);
	RenderTexture->UpdateResourceImmediate();
}

bool USharedRenderTarget::Render(TFunction<void(FCanvas&)> renderFn)
{
	FCanvas canvas(RenderTexture->GameThread_GetRenderTargetResource(), NULL, 0.0f, 0.0f, 0.0f, GMaxRHIFeatureLevel);
    
    renderFn(canvas);
    
    canvas.Flush_GameThread();
    
    return true;
}

Hello theonecalledtom,

Thank you for reporting this issue. Unfortunately, I’m not quite accustomed with accessing pixel data for rendering. Would it be possible for you to provide a sample project that can show the difference in this working/not working on PC / iOS? With a reproduction case I should be able to get this reported and escalated to a developer in this field.

I’ve verified that the RenderTarget has the same pixel format on PC and iOS (PF_FloatRGBA =10). I’ve also tried adding a FlushRenderingCommands prior to calling GameThread_GetRenderTargetResource.

GameThread_GetRenderTargetResource does have a warning on not using the result, though I’ve seen other code in the engine using it.

One thought might be that this needs to be on the render thread not the update thread.

Okay - so I put the following code in my GameMode class:

void AMM_GameMode::Tick(float deltaTime)
{
	Super::Tick(deltaTime);

	if (RenderTexture != nullptr && s_TestTicks<16)
	{
		FCanvas canvas(RenderTexture->GameThread_GetRenderTargetResource(), NULL, 0.0f, 0.0f, 0.0f, GMaxRHIFeatureLevel);
		canvas.Clear(FLinearColor(0.5f, 0.5f, 0.5f, 1.0f));
		canvas.Flush_GameThread(true);
		FlushRenderingCommands();

		TArray<FColor> SurfData;
		auto RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();
		RenderTarget->ReadPixels(SurfData);

		UE_LOG(LogTemp, Display, TEXT("Tick: %d"), s_TestTicks);
		int iToPrint = FMath::Min(4, SurfData.Num());
		for (int i = 0; i < iToPrint; i++)
		{
			auto &c = SurfData[i];
			UE_LOG(LogTemp, Display, TEXT("     {%d, %d, %d, %d}"), c.R, c.G, c.B, c.A);
		}
		++s_TestTicks;
	}

With RenderTexture created in my constructor:

AMM_GameMode::AMM_GameMode(const class FObjectInitializer& PCIP)
: Super(PCIP)
, FontForTextItem(0)
{
	HUDClass = AMM_HUD::StaticClass();

	RenderTexture = PCIP.CreateDefaultSubobject<UTextureRenderTarget2D>(this, TEXT("RenderTexture"));
	RenderTexture->ClearColor = FLinearColor::Black;
	RenderTexture->InitAutoFormat(32, 32);
	RenderTexture->UpdateResourceImmediate();
	s_TestTicks = 0;

And added to my header:

UPROPERTY()
	UTextureRenderTarget2D* RenderTexture = nullptr;

int s_TestTicks = 0;

And it prints different information on iOS and PC. PC is expected, iOS seems like garbage (but seemingly with a pattern to it - more so than I’ve seen in the real code).

file with example project attached - verified on PC though I have yet to get this deployed and tested on iOS.

link text

Confirmed that the test project works (as in demonstrates failure!) on iOS - you’ll have to set the default map to Default and setup your certificates and provisioning appropriately.

Output for me is 188,188,188,255 color values on PC and random on iOS.

Hey - let me know if there is anything more I can provide. In my mind this simple code should provide the same (or very very close) results on the different platforms. If it’s not supported any suggestions on an alternative approach to getting the data would be appreciated.

This functionality is critical to my product shipping and I was hoping to approach a beta release at the end of this month, if there is no engine level solution I can do something with ObjectiveC and the iOS UIImage functionality but it’s going to be ugly, take me a couple of days to implement and I’d prefer not to do it if possible.

I have your project working as expected on Mac - it returns {188, 188, 188, 255} the same as your PC so I’m not sure what’s gone wrong there.

What is clearer is that you’ve found a bug in MetalRHI on iOS. In MetalRHI’s FMetalDynamicRHI::RHIReadSurfaceData the call to the id’s getBytes selector is unsynchronised, so any outstanding render operations that were recently committed may not have completed prior to accessing the internal data. The ‘correct’ way to fix this is to insert a call to Context->SubmitCommandBuffersAndWait() immediately prior to getBytes (and in fact there is one in the Mac specific MTLStorageModeManaged branch, hence why I believe it should work on Mac). If you are building the engine from source you can make that change & rebuild and it should function. If you are using the binary build you can instead change your own code to add a render-thread call to RHIBlockUntilGPUIdle prior to calling FlushRenderThreadCommands & ReadPixels that will serve the same purpose, i.e.:

canvas.Flush_GameThread(true);

ENQUEUE_UNIQUE_RENDER_COMMAND(WaitUntilIdle,

{

GRHICommandList.GetImmediateCommandList().BlockUntilGPUIdle();

});

FlushRenderingCommands();

RenderTarget->ReadPixels(SurfData)

You can use #define’s to eliminate this on other platforms until we fix MetalRHI.

If you are willing to investigate the Mac bug, run your project through Xcode and enable GPU Frame Capture & Metal API Validation in the scheme’s Options panel - that will tell you if something is going wrong with the way we call Metal on Mac.

Thanks for looking into this .

Sadly this change doesn’t seem to fix my test case! My implementation is below - should be exactly what you wrote above but perhaps I missed something?

void AMyGameMode::Tick(float deltaTime)
{
	if (RenderTexture != nullptr && s_TestTicks<16)
	{
		FCanvas canvas(RenderTexture->GameThread_GetRenderTargetResource(), NULL, 0.0f, 0.0f, 0.0f, GMaxRHIFeatureLevel);
		canvas.Clear(FLinearColor(0.5f, 0.5f, 0.5f, 1.0f));
		canvas.Flush_GameThread(true);

#if PLATFORM_IOS
        ENQUEUE_UNIQUE_RENDER_COMMAND(WaitUntilIdle,
                                      {
                                          GRHICommandList.GetImmediateCommandList().BlockUntilGPUIdle();
                                      });
#endif
        FlushRenderingCommands();

		TArray<FColor> SurfData;
		auto RenderTarget = RenderTexture->GameThread_GetRenderTargetResource();
		RenderTarget->ReadPixels(SurfData);

If you can run your project through Xcode with the GPU Frame Capture & Metal API Validation options enabled in the scheme that might tell us why. It is possible that this is something I’ve fixed since 4.12 - when I get a moment I’ll give it a go on an iOS device as well.

Hey , I have those options enabled but don’t see any extra output at the time I’m able to trigger the problem.

I’m an iOS noob though - where would I find the output?

Next question would be should I do something iOS specific or wait for a fix? I’ll probably waste a few days doing something with UIImages - could be less but you know how these things go. I’m probably going to take 4.13 as my last update for a while.

Hello theonecalledtom,

I’m working on reproducing this in 4.12 and then trying in 4.13 to see if it’s been fixed but I wanted to clear something with you first. You mentioned in your previous post that “Output for me is 188,188,188,255 color values on PC and random on iOS.”

I just tried your project in 4.12.0 (Source built from Github) and packaged for iOS through Remote Building on my Windows PC. The result is that for the first 2 ticks the color value doesn’t seem initialized but for the rest of the ticks, everything seems correct. I was loading the Default map if that matters. Here’s the log from that iOS launch. Are you able to see these same results?

P.S. I suggest using Notepad++ for that log file. For some reason iOS logs don’t like to be formatted correctly so it’s atrocious to read in the normal Notepad.

Hi - I should be back on this problem tomorrow - outside chance at tonight, meanwhile though I can confirm that is NOT what I was seeing on iOS in 4.12. I would still count a two frame delay as a problem though one that could be worked around if it were consistent.

My output is more like this - with some variability in just how random the data appears:

2016-07-07 15:18:58.417 MM_04[7026:2217770] [2016.07.07-LogTemp:Display:      {49, 255, 0, 0}
LogTemp:Display:      {49, 255, 255, 0}
LogTemp:Display:      {255, 255, 0, 0}
LogTemp:Display:      {49, 255, 0, 0}

Right now I get hundreds of frames of 0,0,0,0. (I changed the code to remove the N frame test limit).

I’m launching from xcode and looking at the output in xcode.

Thank you for those details. and I are currently looking at the fix he’s working on, as even the results I got before with the uninitialized ticks are unacceptable so we’ll try to get back to you soon with more results.

Is there any fix on this for 4.12? I am having the exact same problem on newer Android devices (older devices seem to work just fine).

Hello NAbreuM,

I apologize for the delay in answering this question, it somehow fell off my list of questions entirely. There are no fixes for this in 4.12 but I did test theonecalledtom’s project in 4.14 and noticed that the output is working perfectly in that version. I would suggest upgrading to 4.14 if you can.

Do I still need to do the

Context->SubmitCommandBuffersAndWait()

in 4.14 or does it just work out of the box?

Shouldn’t be required. I used the project posted in theonecalledtom’s comment from before suggested that without any issues.